Best practice for using Topic in medium to large cluster

[Shimik]

What is the best practice for using topic in medium to large cluster?

In the topologies document (http://activemq.apache.org/topologies.html) it says:
"Multicast is great in development though often you might want to disable this feature in production and have well known servers fixed on specific machines. Often socket based communication (using pointcast) is much faster & better for heavy-lifting - particularly on Java - so we tend to recommend to use multicast mostly for discovery and use TCP / SSL for your heavy duty messaging."

In a system where all the clients need to receive all the messages and some of the messages can be really big (a few megabytes and maybe more)
1. shouldn't multicast be used? What about JGroups or JRMS?
2. Wouldn't it choke the network if I send all the messages to all the clients in TCP?
3. Wouldn't it choke the network having well known servers fixed on specific machines?
4. Does anyone have any experience with this use case?

Shimi

[Shimik]

Does anyone tried this kind of use case with BlobMessage?

Shimi

Shimik wrote:

What is the best practice for using topic in medium to large cluster?

In the topologies document (http://activemq.apache.org/topologies.html) it says:
"Multicast is great in development though often you might want to disable this feature in production and have well known servers fixed on specific machines. Often socket based communication (using pointcast) is much faster & better for heavy-lifting - particularly on Java - so we tend to recommend to use multicast mostly for discovery and use TCP / SSL for your heavy duty messaging."

In a system where all the clients need to receive all the messages and some of the messages can be really big (a few megabytes and maybe more)
1. shouldn't multicast be used? What about JGroups or JRMS?
2. Wouldn't it choke the network if I send all the messages to all the clients in TCP?
3. Wouldn't it choke the network having well known servers fixed on specific machines?
4. Does anyone have any experience with this use case?

Shimi

[James.Strachan]

2008/7/15 Shimik <shimi.k@...>:

>
> What is the best practice for using topic in medium to large cluster?
>
> In the topologies document (http://activemq.apache.org/topologies.html) it
> says:
> "Multicast is great in development though often you might want to disable
> this feature in production and have well known servers fixed on specific
> machines. Often socket based communication (using pointcast) is much faster
> & better for heavy-lifting - particularly on Java - so we tend to recommend
> to use multicast mostly for discovery and use TCP / SSL for your heavy duty
> messaging."
>
> In a system where all the clients need to receive all the messages and some
> of the messages can be really big (a few megabytes and maybe more)
> 1. shouldn't multicast be used? What about JGroups or JRMS?

We recommend TCP on ActiveMQ as (i) we can't use JGroups due to its
license and we don't yet have a solid reliable multicast
implementation.

> 2. Wouldn't it choke the network if I send all the messages to all the
> clients in TCP?

I doubt it; these days you're usually more CPU bound than network
bound if you've a reasonably good network


> 3. Wouldn't it choke the network having well known servers fixed on specific
> machines?

Not necessarily


> 4. Does anyone have any experience with this use case?

Could you give us an idea for how many producers/consumers and what
kinda message throughput rates you're thinking of?


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com

[Shimik]

Although it is not the same product and not the same scale (hopefully it will get to that :-)). Think of a system like youtube where data is big and both the data and traffic keeps on growing. As opposed to youtube (I guess) in this case most of the messages are big 1-10M (can be even more but I expect it to be <5M most of the time) and a good percentage of them needs to get to all of the consumers.

In the beginning there will be 2 producers (for redundancy) and 10 consumers. The system needs to be scalable. I am expecting both traffic and data to grow rapidly. The number of producers will grow with traffic growth and the number of consumers will grow with traffic and data growth.  

James.Strachan wrote:

2008/7/15 Shimik <[e-mail]>:
>
> What is the best practice for using topic in medium to large cluster?
>
> In the topologies document (http://activemq.apache.org/topologies.html) it
> says:
> "Multicast is great in development though often you might want to disable
> this feature in production and have well known servers fixed on specific
> machines. Often socket based communication (using pointcast) is much faster
> & better for heavy-lifting - particularly on Java - so we tend to recommend
> to use multicast mostly for discovery and use TCP / SSL for your heavy duty
> messaging."
>
> In a system where all the clients need to receive all the messages and some
> of the messages can be really big (a few megabytes and maybe more)
> 1. shouldn't multicast be used? What about JGroups or JRMS?

We recommend TCP on ActiveMQ as (i) we can't use JGroups due to its
license and we don't yet have a solid reliable multicast
implementation.

> 2. Wouldn't it choke the network if I send all the messages to all the
> clients in TCP?

I doubt it; these days you're usually more CPU bound than network
bound if you've a reasonably good network


> 3. Wouldn't it choke the network having well known servers fixed on specific
> machines?

Not necessarily


> 4. Does anyone have any experience with this use case?

Could you give us an idea for how many producers/consumers and what
kinda message throughput rates you're thinking of?


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com