Storage algorithm?

[Paul Gale <pa...@gmail.com>]

Hi,

Is there a known algorithm or best practice folks use for determining
values for memory usage, store usage and temp usage for a broker with
a known JVM heap size?

I assume said algorithm would require as input the projected number of
producers and consumers, average message size, message throughput
estimate, number of destinations, etc.

Note: to simplify things we're using only persistent messaging,
durable topic subscribers (no queues) and most producers do not employ
PFC.

Thanks,
Paul

Re: Storage algorithm?

[Paul Gale <pa...@gmail.com>]

>I don't think there is such algorithm. Its best to run load test with your estimated settings and fine tune them if necessary.

I agree. However, what I am looking for specifically is guidance in
the form of what diagnostics to use (other than raw throughput). I
cannot 'fine tune' something unless I have a measurable way of
detecting the effect of each change I make.

>How much there is needed for other tasks depends on the number of clients connected to the broker and the number of destinations. Again, there is no simple formula.

I see where you offered a fellow poster a link to the Fuse Tuning
Guide. This is a start.

Let's simplify and confine things to just memory usage (assuming
persistent messages only, 1200 STOMP producer instances with no PFC).

>memoryUsage should be below the JVM heap size to leave enough heap for other broker tasks that storing messages. How much there is needed for other tasks depends on the number of clients connected to the broker and the number of destinations. Again, there is no simple formula.

Regarding 'enough heap': give me an example of how you would go about
verifying that you've configured this setting appropriately? What
diagnostic(s) would you use? Is there some package specific logging
you would use, for example?

I agree that there's 'no simple formula'. However, there has to be
something more than just 'follow the tuning guide and cross your
fingers that the tests you've executed will suffice.' I know that's
not what you are not saying. However, I would like something a little
less vague.

Personally I believe that if tuning procedures and guidelines were
documented to the extent that they should be that their magnitude
would dwarf the existing documentation. Just my two cents.

Thanks,
Paul

On Tue, Jul 16, 2013 at 3:42 AM, Torsten Mielke <tm...@redhat.com> wrote:
> I don't think there is such algorithm. Its best to run load test with your estimated settings and fine tune them if necessary.
>
> memoryUsage should be below the JVM heap size to leave enough heap for other broker tasks that storing messages. How much there is needed for other tasks depends on the number of clients connected to the broker and the number of destinations. Again, there is no simple formula.
>
> storeUsage can be as big as your disk capacity (or JDBC database capacity) if really needed. To find the appropriate setting you need to ask yourself you man messages you expect your broker to buffer at peak times and what is the average message size.
>
> Similar for temp usage which is used for non-persistent messages.
>
> If you have a scenario where producer and consumer are equally fast and you don't expect messages to pile up in your broker then you can go with fairly low values for each of the settings above.
>
> Hope this helps a bit.
>
>
> Torsten Mielke
> tmielke@redhat.com
> tmielke.blogspot.com
>
>
>
>
>
> On Jul 15, 2013, at 5:26 PM, Paul Gale wrote:
>
>> Hi,
>>
>> Is there a known algorithm or best practice folks use for determining
>> values for memory usage, store usage and temp usage for a broker with
>> a known JVM heap size?
>>
>> I assume said algorithm would require as input the projected number of
>> producers and consumers, average message size, message throughput
>> estimate, number of destinations, etc.
>>
>> Note: to simplify things we're using only persistent messaging,
>> durable topic subscribers (no queues) and most producers do not employ
>> PFC.
>>
>> Thanks,
>> Paul
>

Re: Storage algorithm?

[Torsten Mielke <tm...@redhat.com>]

I don't think there is such algorithm. Its best to run load test with your estimated settings and fine tune them if necessary.

memoryUsage should be below the JVM heap size to leave enough heap for other broker tasks that storing messages. How much there is needed for other tasks depends on the number of clients connected to the broker and the number of destinations. Again, there is no simple formula.

storeUsage can be as big as your disk capacity (or JDBC database capacity) if really needed. To find the appropriate setting you need to ask yourself you man messages you expect your broker to buffer at peak times and what is the average message size.

Similar for temp usage which is used for non-persistent messages.

If you have a scenario where producer and consumer are equally fast and you don't expect messages to pile up in your broker then you can go with fairly low values for each of the settings above. 

Hope this helps a bit.


Torsten Mielke
tmielke@redhat.com
tmielke.blogspot.com





On Jul 15, 2013, at 5:26 PM, Paul Gale wrote:

> Hi,
> 
> Is there a known algorithm or best practice folks use for determining
> values for memory usage, store usage and temp usage for a broker with
> a known JVM heap size?
> 
> I assume said algorithm would require as input the projected number of
> producers and consumers, average message size, message throughput
> estimate, number of destinations, etc.
> 
> Note: to simplify things we're using only persistent messaging,
> durable topic subscribers (no queues) and most producers do not employ
> PFC.
> 
> Thanks,
> Paul