You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Peter Hicks <pe...@poggs.co.uk> on 2019/10/22 17:54:26 UTC
Disk I/O requirement for persistent messages
All,
I have a feed of 110 messages/second of about 150 bytes each which I'm
routing through a default-settings ActiveMQ 5.15.8 server and sending on to
a topic. Everything works fine until I set up a durable subscription, at
which point iostat (Ubuntu 18.04LTS) reports about 300tps and about 2-3
megabytes a second of disk writes, which seems like an awful lot of the
message rate and size, and it's slowing down other processing on the
server. Is this normal and expected?
The server is within Amazon EC2 and I can easily add an additional disk for
the KahaDB directory, but can anyone point me at a resource that will help
me reduce the I/O requirements of persisting all these messages to disk? I
am open to any suggestions.
Peter
--
OpenTrainTimes Ltd. registered in England and Wales, company no.
09504022.
Registered office: 13a Davenant Road, Upper Holloway, London N19
3NW
Re: Disk I/O requirement for persistent messages
Posted by Peter Hicks <pe...@poggs.co.uk>.
Hi Tim, Jean-Baptiste,
I've worked out what the issue is/was:
I have a Camel route which splits groups of 32 elements in a JSON array in
to 32 individual messages. These are small and persistent and so written
to disk when there's a durable subscription in place, causing the high I/O
load.
Even if I have a consumer connected, those messages still appear to be
persisted to disk and then dequeued as they're read - and all the messages
are under 64 bytes. The same happens if I use a queue rather than a topic
and durable subscription. If I consume those messages straight off the
topic without a durable subscription, no problem.
I'm working toward building a test case for this so I can benchmark
strategies for dealing with it - I'll make this available when I've
finished it.
Peter
On Wed, 23 Oct 2019 at 05:51, Tim Bain <tb...@alumni.duke.edu> wrote:
> Peter,
>
> Three quick points to consider:
> 1. It's possible to provision additional IOPS on your EBS volumes to get
> more write throughput, which might be cheaper than EFS. You'd have to run
> the numbers for your particular use case, but it's worth evaluating.
> 2. There are several classes of EBS volume, some based on SSD and some
> based on HDD. Be sure you're using the SSD ones: gp2 or maybe io1 if you
> need the extra performance and the cost numbers work out.
> 3. EBS isn't local. Certain EC2 instance families provide
> physically-attached SSDs, which should be the fastest option available. So
> consider that if you can't get the performance you need out of EBS. But
> you're very vulnerable to data loss in hardware failure situations (or even
> just scaling up the host to a larger instance type), so to go this route
> you need a rock-solid plan for how to manage the data, especially since 5.x
> doesn't provide any means of replicating data between brokers with local
> storage.
>
> Overall, I'm surprised that the performance is as bad as you're describing,
> which sounds like a bug. Would you please write a Bug in JIRA for your
> findings, even if you eventually work around the problems?
>
> Thanks,
> Tim
>
> On Tue, Oct 22, 2019, 10:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Hi,
> >
> > EFS works fine and it's really convenient if you want master/slave
> > setup. The only drawback is that it can costs a lot ;) (I don't if you
> > evaluate the price of using EFS). If you don't need master/slave, EFS is
> > not required (the local fs performance can be good enough depending the
> > kind of EC2 you are using).
> >
> > About KahaDB, with the use case you described, maybe you can try the
> > following configuration:
> >
> > <persistenceAdapter>
> > <kahaDB directory="/path/to/efs"
> > indexWriteBatchSize="1000"
> > indexCacheSize="2000"
> > journalMaxFileLength="32mb"
> > checkForCorruptJournalFiles="false"
> > maxAsyncJobs="5000"
> > concurrentStoreAndDispatchQueues="true"
> > concurrentStoreAndDispatchTopics="true"
> > enableJournalDiskSyncs="true"
> > enableIndexWriteAsync="true"/>
> > </persistenceAdapter>
> >
> > I'm using PostgreSQL with brokers on EC2 today (in production), but even
> > adding some indexes (you can take a look on
> > https://issues.apache.org/jira/browse/AMQ-7008), it's an important
> > bottleneck.
> > I'm now evaluating different alternatives (what I have in mind is to
> > create a network of brokers dynamicOnly).
> > However, it depends a lot of your use case (persistent messages or not,
> > order of messages, etc).
> >
> > Don't hesitate to ping me if you want to discuss about that.
> >
> > Regards
> > JB
> >
> > On 22/10/2019 22:25, Peter Hicks wrote:
> > > Hi JB
> > >
> > > I was using a EBS (local) partition when I saw the horrendous I/O
> > > throughput. During testing over the last hour or so, I've mounted an
> EFS
> > > (for those who don't grok Amazon AWS, Elastic File System: basically an
> > NFS
> > > mount) target and symlinked the kahadb directory over to it.
> > >
> > > My current KahaDB configuration is really nothing special:
> > >
> > > <persistenceAdapter>
> > > <kahaDB directory="${activemq.data}/kahadb"/>
> > > </persistenceAdapter>
> > >
> > > However, the good news is that initial testing using EFS has reduced
> the
> > > load on the server substantially, and the other tasks - in particular,
> a
> > > Camel route which takes a JSON list, converts it to individual messages
> > and
> > > converts each back to a JSON - now run within 5ms, whereas before they
> > were
> > > taking upwards of 1200ms per message.
> > >
> > > I am building a cluster in development, and I'll look at upgrading to
> > > 5.15.9 or .10. Using a JDBC store might be a better fit for me, as I
> > have
> > > plenty of spare capacity on a PostgreSQL server. Is this likely to
> scale
> > > up to a few hundred messages a second, or is KahaDB a better way to go?
> > >
> > >
> > > Peter
> > >
> > >
> > > On Tue, 22 Oct 2019 at 20:20, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > >
> > >> Hi Peter,
> > >>
> > >> The most important is the I/O rate/throughput. I'm also using some
> > >> brokers on EC2 (some using JDBC store, some using kahadb store).
> > >>
> > >> What's the filesystem ? EFS or "local" EC2 ?
> > >>
> > >> What's your current kahadb configuration in activemq.xml ?
> > >>
> > >> Just a note: 5.15.9 got major improvements on kahadb that could help.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On 22/10/2019 19:54, Peter Hicks wrote:
> > >>> All,
> > >>>
> > >>> I have a feed of 110 messages/second of about 150 bytes each which
> I'm
> > >>> routing through a default-settings ActiveMQ 5.15.8 server and sending
> > on
> > >> to
> > >>> a topic. Everything works fine until I set up a durable
> subscription,
> > at
> > >>> which point iostat (Ubuntu 18.04LTS) reports about 300tps and about
> 2-3
> > >>> megabytes a second of disk writes, which seems like an awful lot of
> the
> > >>> message rate and size, and it's slowing down other processing on the
> > >>> server. Is this normal and expected?
> > >>>
> > >>> The server is within Amazon EC2 and I can easily add an additional
> disk
> > >> for
> > >>> the KahaDB directory, but can anyone point me at a resource that will
> > >> help
> > >>> me reduce the I/O requirements of persisting all these messages to
> > >> disk? I
> > >>> am open to any suggestions.
> > >>>
> > >>>
> > >>> Peter
> > >>>
> > >>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbonofre@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
--
OpenTrainTimes Ltd. registered in England and Wales, company no.
09504022.
Registered office: 13a Davenant Road, Upper Holloway, London N19
3NW
Re: Disk I/O requirement for persistent messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Peter,
Three quick points to consider:
1. It's possible to provision additional IOPS on your EBS volumes to get
more write throughput, which might be cheaper than EFS. You'd have to run
the numbers for your particular use case, but it's worth evaluating.
2. There are several classes of EBS volume, some based on SSD and some
based on HDD. Be sure you're using the SSD ones: gp2 or maybe io1 if you
need the extra performance and the cost numbers work out.
3. EBS isn't local. Certain EC2 instance families provide
physically-attached SSDs, which should be the fastest option available. So
consider that if you can't get the performance you need out of EBS. But
you're very vulnerable to data loss in hardware failure situations (or even
just scaling up the host to a larger instance type), so to go this route
you need a rock-solid plan for how to manage the data, especially since 5.x
doesn't provide any means of replicating data between brokers with local
storage.
Overall, I'm surprised that the performance is as bad as you're describing,
which sounds like a bug. Would you please write a Bug in JIRA for your
findings, even if you eventually work around the problems?
Thanks,
Tim
On Tue, Oct 22, 2019, 10:14 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi,
>
> EFS works fine and it's really convenient if you want master/slave
> setup. The only drawback is that it can costs a lot ;) (I don't if you
> evaluate the price of using EFS). If you don't need master/slave, EFS is
> not required (the local fs performance can be good enough depending the
> kind of EC2 you are using).
>
> About KahaDB, with the use case you described, maybe you can try the
> following configuration:
>
> <persistenceAdapter>
> <kahaDB directory="/path/to/efs"
> indexWriteBatchSize="1000"
> indexCacheSize="2000"
> journalMaxFileLength="32mb"
> checkForCorruptJournalFiles="false"
> maxAsyncJobs="5000"
> concurrentStoreAndDispatchQueues="true"
> concurrentStoreAndDispatchTopics="true"
> enableJournalDiskSyncs="true"
> enableIndexWriteAsync="true"/>
> </persistenceAdapter>
>
> I'm using PostgreSQL with brokers on EC2 today (in production), but even
> adding some indexes (you can take a look on
> https://issues.apache.org/jira/browse/AMQ-7008), it's an important
> bottleneck.
> I'm now evaluating different alternatives (what I have in mind is to
> create a network of brokers dynamicOnly).
> However, it depends a lot of your use case (persistent messages or not,
> order of messages, etc).
>
> Don't hesitate to ping me if you want to discuss about that.
>
> Regards
> JB
>
> On 22/10/2019 22:25, Peter Hicks wrote:
> > Hi JB
> >
> > I was using a EBS (local) partition when I saw the horrendous I/O
> > throughput. During testing over the last hour or so, I've mounted an EFS
> > (for those who don't grok Amazon AWS, Elastic File System: basically an
> NFS
> > mount) target and symlinked the kahadb directory over to it.
> >
> > My current KahaDB configuration is really nothing special:
> >
> > <persistenceAdapter>
> > <kahaDB directory="${activemq.data}/kahadb"/>
> > </persistenceAdapter>
> >
> > However, the good news is that initial testing using EFS has reduced the
> > load on the server substantially, and the other tasks - in particular, a
> > Camel route which takes a JSON list, converts it to individual messages
> and
> > converts each back to a JSON - now run within 5ms, whereas before they
> were
> > taking upwards of 1200ms per message.
> >
> > I am building a cluster in development, and I'll look at upgrading to
> > 5.15.9 or .10. Using a JDBC store might be a better fit for me, as I
> have
> > plenty of spare capacity on a PostgreSQL server. Is this likely to scale
> > up to a few hundred messages a second, or is KahaDB a better way to go?
> >
> >
> > Peter
> >
> >
> > On Tue, 22 Oct 2019 at 20:20, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> >> Hi Peter,
> >>
> >> The most important is the I/O rate/throughput. I'm also using some
> >> brokers on EC2 (some using JDBC store, some using kahadb store).
> >>
> >> What's the filesystem ? EFS or "local" EC2 ?
> >>
> >> What's your current kahadb configuration in activemq.xml ?
> >>
> >> Just a note: 5.15.9 got major improvements on kahadb that could help.
> >>
> >> Regards
> >> JB
> >>
> >> On 22/10/2019 19:54, Peter Hicks wrote:
> >>> All,
> >>>
> >>> I have a feed of 110 messages/second of about 150 bytes each which I'm
> >>> routing through a default-settings ActiveMQ 5.15.8 server and sending
> on
> >> to
> >>> a topic. Everything works fine until I set up a durable subscription,
> at
> >>> which point iostat (Ubuntu 18.04LTS) reports about 300tps and about 2-3
> >>> megabytes a second of disk writes, which seems like an awful lot of the
> >>> message rate and size, and it's slowing down other processing on the
> >>> server. Is this normal and expected?
> >>>
> >>> The server is within Amazon EC2 and I can easily add an additional disk
> >> for
> >>> the KahaDB directory, but can anyone point me at a resource that will
> >> help
> >>> me reduce the I/O requirements of persisting all these messages to
> >> disk? I
> >>> am open to any suggestions.
> >>>
> >>>
> >>> Peter
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Disk I/O requirement for persistent messages
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
EFS works fine and it's really convenient if you want master/slave
setup. The only drawback is that it can costs a lot ;) (I don't if you
evaluate the price of using EFS). If you don't need master/slave, EFS is
not required (the local fs performance can be good enough depending the
kind of EC2 you are using).
About KahaDB, with the use case you described, maybe you can try the
following configuration:
<persistenceAdapter>
<kahaDB directory="/path/to/efs"
indexWriteBatchSize="1000"
indexCacheSize="2000"
journalMaxFileLength="32mb"
checkForCorruptJournalFiles="false"
maxAsyncJobs="5000"
concurrentStoreAndDispatchQueues="true"
concurrentStoreAndDispatchTopics="true"
enableJournalDiskSyncs="true"
enableIndexWriteAsync="true"/>
</persistenceAdapter>
I'm using PostgreSQL with brokers on EC2 today (in production), but even
adding some indexes (you can take a look on
https://issues.apache.org/jira/browse/AMQ-7008), it's an important
bottleneck.
I'm now evaluating different alternatives (what I have in mind is to
create a network of brokers dynamicOnly).
However, it depends a lot of your use case (persistent messages or not,
order of messages, etc).
Don't hesitate to ping me if you want to discuss about that.
Regards
JB
On 22/10/2019 22:25, Peter Hicks wrote:
> Hi JB
>
> I was using a EBS (local) partition when I saw the horrendous I/O
> throughput. During testing over the last hour or so, I've mounted an EFS
> (for those who don't grok Amazon AWS, Elastic File System: basically an NFS
> mount) target and symlinked the kahadb directory over to it.
>
> My current KahaDB configuration is really nothing special:
>
> <persistenceAdapter>
> <kahaDB directory="${activemq.data}/kahadb"/>
> </persistenceAdapter>
>
> However, the good news is that initial testing using EFS has reduced the
> load on the server substantially, and the other tasks - in particular, a
> Camel route which takes a JSON list, converts it to individual messages and
> converts each back to a JSON - now run within 5ms, whereas before they were
> taking upwards of 1200ms per message.
>
> I am building a cluster in development, and I'll look at upgrading to
> 5.15.9 or .10. Using a JDBC store might be a better fit for me, as I have
> plenty of spare capacity on a PostgreSQL server. Is this likely to scale
> up to a few hundred messages a second, or is KahaDB a better way to go?
>
>
> Peter
>
>
> On Tue, 22 Oct 2019 at 20:20, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>> Hi Peter,
>>
>> The most important is the I/O rate/throughput. I'm also using some
>> brokers on EC2 (some using JDBC store, some using kahadb store).
>>
>> What's the filesystem ? EFS or "local" EC2 ?
>>
>> What's your current kahadb configuration in activemq.xml ?
>>
>> Just a note: 5.15.9 got major improvements on kahadb that could help.
>>
>> Regards
>> JB
>>
>> On 22/10/2019 19:54, Peter Hicks wrote:
>>> All,
>>>
>>> I have a feed of 110 messages/second of about 150 bytes each which I'm
>>> routing through a default-settings ActiveMQ 5.15.8 server and sending on
>> to
>>> a topic. Everything works fine until I set up a durable subscription, at
>>> which point iostat (Ubuntu 18.04LTS) reports about 300tps and about 2-3
>>> megabytes a second of disk writes, which seems like an awful lot of the
>>> message rate and size, and it's slowing down other processing on the
>>> server. Is this normal and expected?
>>>
>>> The server is within Amazon EC2 and I can easily add an additional disk
>> for
>>> the KahaDB directory, but can anyone point me at a resource that will
>> help
>>> me reduce the I/O requirements of persisting all these messages to
>> disk? I
>>> am open to any suggestions.
>>>
>>>
>>> Peter
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Disk I/O requirement for persistent messages
Posted by Peter Hicks <pe...@poggs.co.uk>.
Hi JB
I was using a EBS (local) partition when I saw the horrendous I/O
throughput. During testing over the last hour or so, I've mounted an EFS
(for those who don't grok Amazon AWS, Elastic File System: basically an NFS
mount) target and symlinked the kahadb directory over to it.
My current KahaDB configuration is really nothing special:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
However, the good news is that initial testing using EFS has reduced the
load on the server substantially, and the other tasks - in particular, a
Camel route which takes a JSON list, converts it to individual messages and
converts each back to a JSON - now run within 5ms, whereas before they were
taking upwards of 1200ms per message.
I am building a cluster in development, and I'll look at upgrading to
5.15.9 or .10. Using a JDBC store might be a better fit for me, as I have
plenty of spare capacity on a PostgreSQL server. Is this likely to scale
up to a few hundred messages a second, or is KahaDB a better way to go?
Peter
On Tue, 22 Oct 2019 at 20:20, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi Peter,
>
> The most important is the I/O rate/throughput. I'm also using some
> brokers on EC2 (some using JDBC store, some using kahadb store).
>
> What's the filesystem ? EFS or "local" EC2 ?
>
> What's your current kahadb configuration in activemq.xml ?
>
> Just a note: 5.15.9 got major improvements on kahadb that could help.
>
> Regards
> JB
>
> On 22/10/2019 19:54, Peter Hicks wrote:
> > All,
> >
> > I have a feed of 110 messages/second of about 150 bytes each which I'm
> > routing through a default-settings ActiveMQ 5.15.8 server and sending on
> to
> > a topic. Everything works fine until I set up a durable subscription, at
> > which point iostat (Ubuntu 18.04LTS) reports about 300tps and about 2-3
> > megabytes a second of disk writes, which seems like an awful lot of the
> > message rate and size, and it's slowing down other processing on the
> > server. Is this normal and expected?
> >
> > The server is within Amazon EC2 and I can easily add an additional disk
> for
> > the KahaDB directory, but can anyone point me at a resource that will
> help
> > me reduce the I/O requirements of persisting all these messages to
> disk? I
> > am open to any suggestions.
> >
> >
> > Peter
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
OpenTrainTimes Ltd. registered in England and Wales, company no.
09504022.
Registered office: 13a Davenant Road, Upper Holloway, London N19
3NW
Re: Disk I/O requirement for persistent messages
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Peter,
The most important is the I/O rate/throughput. I'm also using some
brokers on EC2 (some using JDBC store, some using kahadb store).
What's the filesystem ? EFS or "local" EC2 ?
What's your current kahadb configuration in activemq.xml ?
Just a note: 5.15.9 got major improvements on kahadb that could help.
Regards
JB
On 22/10/2019 19:54, Peter Hicks wrote:
> All,
>
> I have a feed of 110 messages/second of about 150 bytes each which I'm
> routing through a default-settings ActiveMQ 5.15.8 server and sending on to
> a topic. Everything works fine until I set up a durable subscription, at
> which point iostat (Ubuntu 18.04LTS) reports about 300tps and about 2-3
> megabytes a second of disk writes, which seems like an awful lot of the
> message rate and size, and it's slowing down other processing on the
> server. Is this normal and expected?
>
> The server is within Amazon EC2 and I can easily add an additional disk for
> the KahaDB directory, but can anyone point me at a resource that will help
> me reduce the I/O requirements of persisting all these messages to disk? I
> am open to any suggestions.
>
>
> Peter
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com