You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/03/13 04:00:10 UTC
Lots of small ActiveMQ instances or one big one?
I’m trying to improve our ActiveMQ message throughput and I’m not really
seeing the performance I would expect.
We moved from non-persistent to persistent (which would be more ideal) and
the performance is about 4-5x slower than before.
I think this would be somewhat reasonable but our disks aren’t really at
100% utilization and they’re on SSD. They’re only at about 5% utilization.
So perhaps instead of ONE ActiveMQ node on a 16 core box it might be more
beneficial to use say 4 or 8 ? (maybe putting them in containers).
Granted this would require a TON of work right now, but that might be the
best way to go (in theory).
Kevin
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Kevin Burton <bu...@spinn3r.com>.
Yes. I’m curious about this as well. I can’t easily benchmark / profile
this box to find the lock.
I just did some analysis and it looks like LevelDB imposed about a 2x
overhead just in terms of CPU.
I setup a benchmark backed by a RAM disk and LevelDB and this performed 2x
slower than just a non-persistent broker.
It isn’t exactly apples to apples but I think it’s close.
I also identified a bottleneck in my threading code sharing a connection so
that’s an issue too.
On Thu, Mar 12, 2015 at 8:38 PM, Tim Bain <tb...@alumni.duke.edu> wrote:
> Keep in mind that as soon as you have a network of brokers, some portion of
> your messages will have to cross two brokers to get from producer to
> consumer (and more than that if your topology isn't a mesh) instead of just
> one. So my gut instinct (with nothing to back it up) is that if you're
> going to subdivide, you should break up into as many instances as you think
> your server can handle; definitely don't just make a cluster of 2 brokers,
> or you might end up slower than you were with just one. Then you just need
> to make sure you do a good job of balancing your producers and your
> consumers across the cluster (which might be easy or might be hard,
> depending on how regular/predictable their workloads are).
>
> But I'm curious about what your bottleneck is if it's not disk I/O. Are
> you pegging CPUs? Are you saturating your NICs? Madly swapping memory?
> If none of the above, maybe locking within the KahaDB/LevelDB (which is
> it?) code is to blame...
>
> Tim
>
> On Thu, Mar 12, 2015 at 9:00 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
> > I’m trying to improve our ActiveMQ message throughput and I’m not really
> > seeing the performance I would expect.
> >
> > We moved from non-persistent to persistent (which would be more ideal)
> and
> > the performance is about 4-5x slower than before.
> >
> > I think this would be somewhat reasonable but our disks aren’t really at
> > 100% utilization and they’re on SSD. They’re only at about 5%
> utilization.
> >
> > So perhaps instead of ONE ActiveMQ node on a 16 core box it might be more
> > beneficial to use say 4 or 8 ? (maybe putting them in containers).
> >
> > Granted this would require a TON of work right now, but that might be the
> > best way to go (in theory).
> >
> > Kevin
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Tim Bain <tb...@alumni.duke.edu>.
Keep in mind that as soon as you have a network of brokers, some portion of
your messages will have to cross two brokers to get from producer to
consumer (and more than that if your topology isn't a mesh) instead of just
one. So my gut instinct (with nothing to back it up) is that if you're
going to subdivide, you should break up into as many instances as you think
your server can handle; definitely don't just make a cluster of 2 brokers,
or you might end up slower than you were with just one. Then you just need
to make sure you do a good job of balancing your producers and your
consumers across the cluster (which might be easy or might be hard,
depending on how regular/predictable their workloads are).
But I'm curious about what your bottleneck is if it's not disk I/O. Are
you pegging CPUs? Are you saturating your NICs? Madly swapping memory?
If none of the above, maybe locking within the KahaDB/LevelDB (which is
it?) code is to blame...
Tim
On Thu, Mar 12, 2015 at 9:00 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I’m trying to improve our ActiveMQ message throughput and I’m not really
> seeing the performance I would expect.
>
> We moved from non-persistent to persistent (which would be more ideal) and
> the performance is about 4-5x slower than before.
>
> I think this would be somewhat reasonable but our disks aren’t really at
> 100% utilization and they’re on SSD. They’re only at about 5% utilization.
>
> So perhaps instead of ONE ActiveMQ node on a 16 core box it might be more
> beneficial to use say 4 or 8 ? (maybe putting them in containers).
>
> Granted this would require a TON of work right now, but that might be the
> best way to go (in theory).
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Kevin Burton <bu...@spinn3r.com>.
Keep in mind though that if you do
> 10ms batches, you're going to add an average of 5ms latency to your
> messages compared to the theoretical throughput of a SSD that didn't have
> that limitation, even if your current message rates are slow enough that
> the SSD can keep up without batching.
>
>
Oh. That’s one of the cool things of the ’smart batching’ algorithm. The
latencies will fall on SSD automatically.
This is because we just grab ALL the messages (or a configured limit of
them) from the queue, then commit them all at once, then bulk ack them.
So in practice , on SSD, the latencies would be a function of the IOPS of
the drive.
So if the SSD can do 10k IOPS , the packet latency will be around 100us..
ALL the messages on the queue is probably a bit extreme. You wouldn’t want
to read too much because that would increase latency. So maybe a
maxSmartBatchSize would be good.
So if batching is implemented, I'd definitely want it to be a configurable
> option rather than the one and only way it works.
Agreed. It would probably need to be implemented this way just to avoid
scaring people. If it works well it could be enabled by default in a
future implementation.
> The easy option would
> simply be a configuration option to turn batching on or off (and always pay
> the 5ms latency cost); the better option would be an adaptive strategy that
> (if enabled) recognizes when you throughput crosses the worth-it threshold
> and begins batching, and then recognizes when you drop back below it and
> goes back to the sync-per-message approach.
>
>
Note that batching isn’t enabled if you’re writing at a slow rate.
If you emit one message per second, the queue length per commit will be 1
so effectively you’re batching with 1 item in the batch. so the latency
isn’t there.
> Also, keep in mind that this strategy is only going to make a performance
> improvement if it's possible to process messages in parallel; if the code
> is written so you can't move to the next message in a destination till the
> previous one is sync'ed, then you'll get an improvement by eliminating
> contention between destinations but you'll go the same speed (slower,
> actually, due to the 5ms average latency of the batching) within a single
> destination. So as usual, YMMV, which is another reason why this should be
> an option rather than the only option.
>
Agreed. I started looking at the code and it wasn’t immediately obvious.
I’d probably need 2-4 hours and I just don’t have it right now :-(
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Tim Bain <tb...@alumni.duke.edu>.
I actually didn't realize that HDDs had that performance limitation, so
thanks for that info, and it now makes sense why in certain circumstances
(i.e. where messages are more closely spaced than 10ms/msg) there could be
a throughput improvement by batching. Keep in mind though that if you do
10ms batches, you're going to add an average of 5ms latency to your
messages compared to the theoretical throughput of a SSD that didn't have
that limitation, even if your current message rates are slow enough that
the SSD can keep up without batching.
So if batching is implemented, I'd definitely want it to be a configurable
option rather than the one and only way it works. The easy option would
simply be a configuration option to turn batching on or off (and always pay
the 5ms latency cost); the better option would be an adaptive strategy that
(if enabled) recognizes when you throughput crosses the worth-it threshold
and begins batching, and then recognizes when you drop back below it and
goes back to the sync-per-message approach.
Also, keep in mind that this strategy is only going to make a performance
improvement if it's possible to process messages in parallel; if the code
is written so you can't move to the next message in a destination till the
previous one is sync'ed, then you'll get an improvement by eliminating
contention between destinations but you'll go the same speed (slower,
actually, due to the 5ms average latency of the batching) within a single
destination. So as usual, YMMV, which is another reason why this should be
an option rather than the only option.
Tim
On Sat, Mar 14, 2015 at 12:35 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> > So if you batch disk writes, you either batch acks (which has a negative
> > performance impact; I'm not sure if the loss will be greater or less than
> > the gain from batching writes, but at a minimum it'll decrease the
> expected
> > improvement from batching writes)
>
>
>
> Yes. This is the point I’m trying to make.
>
> If you’re on HDD without any form of write caching, the MAX you can commit
> is around 100 sync’s per second.
>
> So the sync throughput for ActiveMQ is lower than this due to ActiveMQ
> needing to do additional work (memory copying, etc). But in theory it’s
> probably 95% of this number.
>
> However, if you batch commit, you have 10ms between each sync where you can
> accumulate messages. ALL the messages you accumulate in that window can
> then be sync’d in the next round.
>
> (I’m sure you guys know this stuff, just codifying it)
>
> The next time around you can write ALL the message at once, and with
> LevelDB the performance should be near sequential disk IO (in theory,
> since it’s writing to the log) which is pretty darn fast.
>
> So in practice you DO have to batch the ACKs back but that’s more than made
> up for with the performance bump.
>
> Disk is just so amazingly slow.
>
> I’d be willing to do a proof of concept branch with benchmarks if someone
> can point me to the right spot in the code. Should be a small chunk of
> code.
>
> or you optimistically ack before the bits
> > hit the disk (which violates the JMS contract because it allows message
> > loss if the server crashes before the batch is written).
> >
> >
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Kevin Burton <bu...@spinn3r.com>.
> So if you batch disk writes, you either batch acks (which has a negative
> performance impact; I'm not sure if the loss will be greater or less than
> the gain from batching writes, but at a minimum it'll decrease the expected
> improvement from batching writes)
Yes. This is the point I’m trying to make.
If you’re on HDD without any form of write caching, the MAX you can commit
is around 100 sync’s per second.
So the sync throughput for ActiveMQ is lower than this due to ActiveMQ
needing to do additional work (memory copying, etc). But in theory it’s
probably 95% of this number.
However, if you batch commit, you have 10ms between each sync where you can
accumulate messages. ALL the messages you accumulate in that window can
then be sync’d in the next round.
(I’m sure you guys know this stuff, just codifying it)
The next time around you can write ALL the message at once, and with
LevelDB the performance should be near sequential disk IO (in theory,
since it’s writing to the log) which is pretty darn fast.
So in practice you DO have to batch the ACKs back but that’s more than made
up for with the performance bump.
Disk is just so amazingly slow.
I’d be willing to do a proof of concept branch with benchmarks if someone
can point me to the right spot in the code. Should be a small chunk of
code.
or you optimistically ack before the bits
> hit the disk (which violates the JMS contract because it allows message
> loss if the server crashes before the batch is written).
>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Christian Grassi <ch...@gmail.com>.
Yes Tim, this was my point, if the producer receives the ack, it should be
when the message is persistet on disk.
Chris
Il giorno sab 14 mar 2015 15:32 Tim Bain <tb...@alumni.duke.edu> ha scritto:
> I think the difference is that an ack packet (telling the sender that it
> can discard the message because the broker now has it stored and the
> sender's copy is no longer needed as a backup) is sent after a successful
> write to disk, so the consequences of a failed disk write are different
> than a failed Ethernet or queue write.
>
> So if you batch disk writes, you either batch acks (which has a negative
> performance impact; I'm not sure if the loss will be greater or less than
> the gain from batching writes, but at a minimum it'll decrease the expected
> improvement from batching writes) or you optimistically ack before the bits
> hit the disk (which violates the JMS contract because it allows message
> loss if the server crashes before the batch is written).
> On Mar 13, 2015 7:57 PM, "Kevin Burton" <bu...@spinn3r.com> wrote:
>
> > > Regarding batching in my opinion it is right, even if is not the most
> > performing solution, to have a fsync per msg as otherwise in case of
> outage
> > you lose more messages
> >
> > To clarify, you get the same mathematical / functional properties of
> > messages, just in the case of smart batching, you get MUCH better
> > throughput.
> >
> > While the packet is over the ethernet it’s not sync’d .. while it’s
> waiting
> > in queue, it’s not sync’d.
> >
> > The change is that instead of taking one message at a time, and writing,
> > then syncing each one, you either read N messages at once, or the entire
> > queue.
> >
> > The performance improvement can be dramatic. (100x).
> >
> >
> > On Fri, Mar 13, 2015 at 5:59 PM, Christian Grassi <
> > christiangrassi@gmail.com
> > > wrote:
> >
> > > Hi Kevin,
> > > just curios how many messages per second you are seeing?
> > > Regarding batching in my opinion it is right, even if is not the most
> > > performing solution, to have a fsync per msg as otherwise in case of
> > outage
> > > you lose more messages. Personally speaking in a persistent queue if
> the
> > > producer is acknowledged that a message is in the queue it should be
> sent
> > > and persisted.
> > > Just my opinion
> > >
> > > Chris
> > >
> > > Il giorno ven 13 mar 2015 23:06 Kevin Burton <bu...@spinn3r.com> ha
> > > scritto:
> > >
> > > > Does ActiveMQ do an fsync per message?
> > > >
> > > > Why not do smart batching:
> > > >
> > > > http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
> > > >
> > > > This way the messages could be elided, and one fsync called on all of
> > > them.
> > > >
> > > > Maybe it already does, but the comment that each messages requires a
> > sync
> > > > seems to hint that this could be more efficient.
> > > >
> > > > On Fri, Mar 13, 2015 at 2:46 PM, artnaseef <ar...@artnaseef.com>
> wrote:
> > > >
> > > > > I think they are in-line. More producers generally mean more
> > > concurrent
> > > > > work
> > > > > for the broker, as a single producer sends all of its messages
> > serially
> > > > to
> > > > > the broker. Similar logic applies to consumers.
> > > > >
> > > > > More destinations generally forces more producers and consumers,
> > > although
> > > > > it
> > > > > doesn't have to.
> > > > >
> > > > > Is there something more specific that doesn't seem to match?
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > View this message in context:
> > > > > http://activemq.2283324.n4.nabble.com/Lots-of-small-
> > > > ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
> > > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Founder/CEO Spinn3r.com
> > > > Location: *San Francisco, CA*
> > > > blog: http://burtonator.wordpress.com
> > > > … or check out my Google+ profile
> > > > <https://plus.google.com/102718274791889610666/posts>
> > > > <http://spinn3r.com>
> > > >
> > >
> >
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Tim Bain <tb...@alumni.duke.edu>.
I think the difference is that an ack packet (telling the sender that it
can discard the message because the broker now has it stored and the
sender's copy is no longer needed as a backup) is sent after a successful
write to disk, so the consequences of a failed disk write are different
than a failed Ethernet or queue write.
So if you batch disk writes, you either batch acks (which has a negative
performance impact; I'm not sure if the loss will be greater or less than
the gain from batching writes, but at a minimum it'll decrease the expected
improvement from batching writes) or you optimistically ack before the bits
hit the disk (which violates the JMS contract because it allows message
loss if the server crashes before the batch is written).
On Mar 13, 2015 7:57 PM, "Kevin Burton" <bu...@spinn3r.com> wrote:
> > Regarding batching in my opinion it is right, even if is not the most
> performing solution, to have a fsync per msg as otherwise in case of outage
> you lose more messages
>
> To clarify, you get the same mathematical / functional properties of
> messages, just in the case of smart batching, you get MUCH better
> throughput.
>
> While the packet is over the ethernet it’s not sync’d .. while it’s waiting
> in queue, it’s not sync’d.
>
> The change is that instead of taking one message at a time, and writing,
> then syncing each one, you either read N messages at once, or the entire
> queue.
>
> The performance improvement can be dramatic. (100x).
>
>
> On Fri, Mar 13, 2015 at 5:59 PM, Christian Grassi <
> christiangrassi@gmail.com
> > wrote:
>
> > Hi Kevin,
> > just curios how many messages per second you are seeing?
> > Regarding batching in my opinion it is right, even if is not the most
> > performing solution, to have a fsync per msg as otherwise in case of
> outage
> > you lose more messages. Personally speaking in a persistent queue if the
> > producer is acknowledged that a message is in the queue it should be sent
> > and persisted.
> > Just my opinion
> >
> > Chris
> >
> > Il giorno ven 13 mar 2015 23:06 Kevin Burton <bu...@spinn3r.com> ha
> > scritto:
> >
> > > Does ActiveMQ do an fsync per message?
> > >
> > > Why not do smart batching:
> > >
> > > http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
> > >
> > > This way the messages could be elided, and one fsync called on all of
> > them.
> > >
> > > Maybe it already does, but the comment that each messages requires a
> sync
> > > seems to hint that this could be more efficient.
> > >
> > > On Fri, Mar 13, 2015 at 2:46 PM, artnaseef <ar...@artnaseef.com> wrote:
> > >
> > > > I think they are in-line. More producers generally mean more
> > concurrent
> > > > work
> > > > for the broker, as a single producer sends all of its messages
> serially
> > > to
> > > > the broker. Similar logic applies to consumers.
> > > >
> > > > More destinations generally forces more producers and consumers,
> > although
> > > > it
> > > > doesn't have to.
> > > >
> > > > Is there something more specific that doesn't seem to match?
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > > http://activemq.2283324.n4.nabble.com/Lots-of-small-
> > > ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Founder/CEO Spinn3r.com
> > > Location: *San Francisco, CA*
> > > blog: http://burtonator.wordpress.com
> > > … or check out my Google+ profile
> > > <https://plus.google.com/102718274791889610666/posts>
> > > <http://spinn3r.com>
> > >
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Kevin Burton <bu...@spinn3r.com>.
> Regarding batching in my opinion it is right, even if is not the most
performing solution, to have a fsync per msg as otherwise in case of outage
you lose more messages
To clarify, you get the same mathematical / functional properties of
messages, just in the case of smart batching, you get MUCH better
throughput.
While the packet is over the ethernet it’s not sync’d .. while it’s waiting
in queue, it’s not sync’d.
The change is that instead of taking one message at a time, and writing,
then syncing each one, you either read N messages at once, or the entire
queue.
The performance improvement can be dramatic. (100x).
On Fri, Mar 13, 2015 at 5:59 PM, Christian Grassi <christiangrassi@gmail.com
> wrote:
> Hi Kevin,
> just curios how many messages per second you are seeing?
> Regarding batching in my opinion it is right, even if is not the most
> performing solution, to have a fsync per msg as otherwise in case of outage
> you lose more messages. Personally speaking in a persistent queue if the
> producer is acknowledged that a message is in the queue it should be sent
> and persisted.
> Just my opinion
>
> Chris
>
> Il giorno ven 13 mar 2015 23:06 Kevin Burton <bu...@spinn3r.com> ha
> scritto:
>
> > Does ActiveMQ do an fsync per message?
> >
> > Why not do smart batching:
> >
> > http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
> >
> > This way the messages could be elided, and one fsync called on all of
> them.
> >
> > Maybe it already does, but the comment that each messages requires a sync
> > seems to hint that this could be more efficient.
> >
> > On Fri, Mar 13, 2015 at 2:46 PM, artnaseef <ar...@artnaseef.com> wrote:
> >
> > > I think they are in-line. More producers generally mean more
> concurrent
> > > work
> > > for the broker, as a single producer sends all of its messages serially
> > to
> > > the broker. Similar logic applies to consumers.
> > >
> > > More destinations generally forces more producers and consumers,
> although
> > > it
> > > doesn't have to.
> > >
> > > Is there something more specific that doesn't seem to match?
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > > http://activemq.2283324.n4.nabble.com/Lots-of-small-
> > ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Christian Grassi <ch...@gmail.com>.
Hi Kevin,
just curios how many messages per second you are seeing?
Regarding batching in my opinion it is right, even if is not the most
performing solution, to have a fsync per msg as otherwise in case of outage
you lose more messages. Personally speaking in a persistent queue if the
producer is acknowledged that a message is in the queue it should be sent
and persisted.
Just my opinion
Chris
Il giorno ven 13 mar 2015 23:06 Kevin Burton <bu...@spinn3r.com> ha
scritto:
> Does ActiveMQ do an fsync per message?
>
> Why not do smart batching:
>
> http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
>
> This way the messages could be elided, and one fsync called on all of them.
>
> Maybe it already does, but the comment that each messages requires a sync
> seems to hint that this could be more efficient.
>
> On Fri, Mar 13, 2015 at 2:46 PM, artnaseef <ar...@artnaseef.com> wrote:
>
> > I think they are in-line. More producers generally mean more concurrent
> > work
> > for the broker, as a single producer sends all of its messages serially
> to
> > the broker. Similar logic applies to consumers.
> >
> > More destinations generally forces more producers and consumers, although
> > it
> > doesn't have to.
> >
> > Is there something more specific that doesn't seem to match?
> >
> >
> >
> > --
> > View this message in context:
> > http://activemq.2283324.n4.nabble.com/Lots-of-small-
> ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Kevin Burton <bu...@spinn3r.com>.
Does ActiveMQ do an fsync per message?
Why not do smart batching:
http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html
This way the messages could be elided, and one fsync called on all of them.
Maybe it already does, but the comment that each messages requires a sync
seems to hint that this could be more efficient.
On Fri, Mar 13, 2015 at 2:46 PM, artnaseef <ar...@artnaseef.com> wrote:
> I think they are in-line. More producers generally mean more concurrent
> work
> for the broker, as a single producer sends all of its messages serially to
> the broker. Similar logic applies to consumers.
>
> More destinations generally forces more producers and consumers, although
> it
> doesn't have to.
>
> Is there something more specific that doesn't seem to match?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Lots-of-small-ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by artnaseef <ar...@artnaseef.com>.
I think they are in-line. More producers generally mean more concurrent work
for the broker, as a single producer sends all of its messages serially to
the broker. Similar logic applies to consumers.
More destinations generally forces more producers and consumers, although it
doesn't have to.
Is there something more specific that doesn't seem to match?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Lots-of-small-ActiveMQ-instances-or-one-big-one-tp4693133p4693203.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Lots of small ActiveMQ instances or one big one?
Posted by Kevin Burton <bu...@spinn3r.com>.
I’m getting mixed messages :-P
This statement:
To drive the disk, the more parallel work that the brokers can
> introduce the better. In other words, to get max throughput, you can
> increase the num of parallel jms connections per destination, the the
> number of destinations per broker, then the number of brokers. At each
> stage track peek throughput till it starts to drop off before
> introducing the next level of parallelism.
conflicts with this one:
Increasing the number of producers and destinations involved will increase the
> disk utilization.
Unless I’m missing something.
I have some tests that show that the more workers the better throughput.
On Fri, Mar 13, 2015 at 1:30 PM, artnaseef <ar...@artnaseef.com> wrote:
> Exactly where my thoughts were going. The usage of destinations will
> have a
> big impact here. If there is one producer to a single queue with
> persistent
> messages, the bottleneck is going to be the message acknowledgements which
> require the "sync" to disk.
>
> Increasing the number of producers and destinations involved will increase
> the disk utilization.
>
> Syncs are hard to see in the scheme of things since they involve waits at
> several points and don't show up as significant CPU usage, disk usage, etc.
> In other words, it can be hard to see that syncs are the bottleneck.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Lots-of-small-ActiveMQ-instances-or-one-big-one-tp4693133p4693192.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by artnaseef <ar...@artnaseef.com>.
Exactly where my thoughts were going. The usage of destinations will have a
big impact here. If there is one producer to a single queue with persistent
messages, the bottleneck is going to be the message acknowledgements which
require the "sync" to disk.
Increasing the number of producers and destinations involved will increase
the disk utilization.
Syncs are hard to see in the scheme of things since they involve waits at
several points and don't show up as significant CPU usage, disk usage, etc.
In other words, it can be hard to see that syncs are the bottleneck.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Lots-of-small-ActiveMQ-instances-or-one-big-one-tp4693133p4693192.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Lots of small ActiveMQ instances or one big one?
Posted by artnaseef <ar...@artnaseef.com>.
+1
Ditto
--
View this message in context: http://activemq.2283324.n4.nabble.com/Lots-of-small-ActiveMQ-instances-or-one-big-one-tp4693133p4693193.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Lots of small ActiveMQ instances or one big one?
Posted by Gary Tully <ga...@gmail.com>.
and +1 to what Tim says ;-)
On 13 March 2015 at 12:17, Gary Tully <ga...@gmail.com> wrote:
> kevin - peek at https://issues.apache.org/jira/browse/AMQ-5578 it
> gives a big win in terms of OS system work per journal write, it is
> however kahadb focused at the moment.
>
> To drive the disk, the more parallel work that the brokers can
> introduce the better. In other words, to get max throughput, you can
> increase the num of parallel jms connections per destination, the the
> number of destinations per broker, then the number of brokers. At each
> stage track peek throughput till it starts to drop off before
> introducing the next level of parallelism.
>
> A broker will try and batch writes but it needs parallel connections
> and or destinations to do that.
>
> With SSD, write performance may not be the bottleneck, but working
> through a process like the above will identify what is.
>
> On 13 March 2015 at 03:00, Kevin Burton <bu...@spinn3r.com> wrote:
>> I’m trying to improve our ActiveMQ message throughput and I’m not really
>> seeing the performance I would expect.
>>
>> We moved from non-persistent to persistent (which would be more ideal) and
>> the performance is about 4-5x slower than before.
>>
>> I think this would be somewhat reasonable but our disks aren’t really at
>> 100% utilization and they’re on SSD. They’re only at about 5% utilization.
>>
>> So perhaps instead of ONE ActiveMQ node on a 16 core box it might be more
>> beneficial to use say 4 or 8 ? (maybe putting them in containers).
>>
>> Granted this would require a TON of work right now, but that might be the
>> best way to go (in theory).
>>
>> Kevin
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
Re: Lots of small ActiveMQ instances or one big one?
Posted by Gary Tully <ga...@gmail.com>.
kevin - peek at https://issues.apache.org/jira/browse/AMQ-5578 it
gives a big win in terms of OS system work per journal write, it is
however kahadb focused at the moment.
To drive the disk, the more parallel work that the brokers can
introduce the better. In other words, to get max throughput, you can
increase the num of parallel jms connections per destination, the the
number of destinations per broker, then the number of brokers. At each
stage track peek throughput till it starts to drop off before
introducing the next level of parallelism.
A broker will try and batch writes but it needs parallel connections
and or destinations to do that.
With SSD, write performance may not be the bottleneck, but working
through a process like the above will identify what is.
On 13 March 2015 at 03:00, Kevin Burton <bu...@spinn3r.com> wrote:
> I’m trying to improve our ActiveMQ message throughput and I’m not really
> seeing the performance I would expect.
>
> We moved from non-persistent to persistent (which would be more ideal) and
> the performance is about 4-5x slower than before.
>
> I think this would be somewhat reasonable but our disks aren’t really at
> 100% utilization and they’re on SSD. They’re only at about 5% utilization.
>
> So perhaps instead of ONE ActiveMQ node on a 16 core box it might be more
> beneficial to use say 4 or 8 ? (maybe putting them in containers).
>
> Granted this would require a TON of work right now, but that might be the
> best way to go (in theory).
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>