You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Albert Strasheim <13...@sun.ac.za> on 2007/01/26 17:44:05 UTC
Fast producer, slow consumer with spooling to disk?
Hello all
Following on from these threads:
http://www.nabble.com/Blocking-on-UsageManager.waitForSpace-again-tf3031460.html
http://www.nabble.com/OutOfMemoryErrors-again-tf3083798.html
I've updated my ActiveMQ 4.2 from trunk to include Rob's latest
patches, but I'm still having problems.
Looking at the example I posted here:
http://www.nabble.com/Re%3A-Blocking-on-UsageManager.waitForSpace-again-p8521205.html
is there a way in a fast producer, slow consumer scenario to configure
ActiveMQ to block the producer until space becomes available for it to
send more messages, without spooling to disk or running out of memory
(maybe with some kind of configurable timeout on how long the producer
blocks before throwing)? We want the messages to move at the speed of
the slowest consumer (since we only ever have one consumer), not the
producer.
Running that example, with the Sun JDK's default heap size, I still
run into OutOfMemoryErrors. Increasing the heap size with -Xms256m
-Xmx512m causes the program to run, until it promptly fills up the
disk on my machine. I've specified both broker.persistent=false and
DeliveryMode.NON_PERSISTENT on the producer, but it seems some kind of
persistence is still active, since I promptly see my activemq-data
directory filling up.
Thanks!
Cheers,
Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by yaussy <ya...@cboe.com>.
Rob,
This goes back to some conversations I've had with you concerning durable
messages. I'd like the opposite: I don't want the publisher to be blocked,
and I don't want the Broker to queue in memory when the consumer is slow.
I'd like to have the Broker queue to disk.
I've used the "keepDurableSubsActive" switch on the broker to handle a
consumer that is no longer running. That works fine, but there's still the
issue of the slow consumer. I'd started to work with AMQ 4.1, but still saw
the same behavior as before. I've been busy with some other things, so
haven't got back to looking at recent code, i.e. 4.2. So, in 4.2, do you
have something that will give the behavior we'd like to see?
Kevin
rajdavies wrote:
>
> Hi Albert,
>
> from the latest in SVN it's possible to configure the cursor used for
> a topic subscriber - which by default is file based. If you use a VM
> based cursor, as messages aren't removed from memory (to spool to
> disk) - the producer should block. You can configure the cursor type
> using destination policies: http://incubator.apache.org/activemq/
> xbean-xml-reference-41.html#XBeanXMLReference4.1-policyEntryelement
>
>
> An example is:
>
> <beans>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfi
> gurer"/>
>
> <broker persistent="false" xmlns="http://activemq.org/config/1.0">
>
> <!-- lets define the dispatch policy -->
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic="org.apache.>">
> <dispatchPolicy>
> <strictOrderDispatchPolicy />
> </dispatchPolicy>
> <deadLetterStrategy>
> <individualDeadLetterStrategy topicPrefix="Test.DLQ." />
> </deadLetterStrategy>
> <pendingSubscriberPolicy>
> <vmCursor />
> </pendingSubscriberPolicy>
> </policyEntry>
>
> <policyEntry queue="org.apache.>">
> <dispatchPolicy>
> <strictOrderDispatchPolicy />
> </dispatchPolicy>
> <deadLetterStrategy>
> <individualDeadLetterStrategy queuePrefix="Test.DLQ."/>
> </deadLetterStrategy>
> <pendingQueuePolicy>
> <vmQueueCursor />
> </pendingQueuePolicy>
> </policyEntry>
>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> </broker>
>
> </beans>
>
> where the all topic subscribers to destinations starting with
> org.apache will have a VMCursor for their pending messages.
>
> hope that helps
>
> cheers,
>
> Rob
>
> On 26 Jan 2007, at 17:53, Albert Strasheim wrote:
>
>> Hey Rob,
>>
>> On Fri, 26 Jan 2007, Rob Davies wrote:
>>
>>> Hi Albert,
>>> I think you could need to set the prefetch limit as well - it could
>>> be the clients running out of memory.
>>
>> Thanks, I'll experiment with the prefetch limit a bit.
>>
>>> I agree we need to allow for configuration of blocking a producer
>>> whilst waiting for space - be that disk or memory - this may take a
>>> few more days to get right :(
>>
>> Will you have time to look into this? Could you provide some hints
>> as to
>> where one should start looking to implement this feature.
>>
>> Thanks for the quick reply and the patches so far!
>>
>> Cheers,
>>
>> Albert
>
>
>
--
View this message in context: http://www.nabble.com/Fast-producer%2C-slow-consumer-with-spooling-to-disk--tf3123868.html#a8696663
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Fast producer, slow consumer with spooling to disk?
Posted by Albert Strasheim <13...@sun.ac.za>.
Hello all,
On Fri, 02 Feb 2007, Rob Davies wrote:
> Hi Albert,
>
> what is the behavior you are expecting ?
>
> cheers,
>
> Rob
Hey Rob. What we'd like to happen in our fast producer/slow consumer
scenario when the broker runs out of space (in memory) is that it
blocks the producers for a while, letting the slow consumers consume
some messages to free up space for the producers to send more.
We have this scenario in cases where our producer is something that
streams large amounts of data from, e.g., disk/database/video capture
card to a consumer that takes long to process each message (e.g.,
running a bunch of image processing algorithms).
In the case where one streams from something like a video capture card,
you'll want a policy that starts dropping frames after they haven't been
consumed for a while. In the database case (say you recorded some video
on another machine), you don't really want to drop these frames -- as
long as the consumer keeps ticking over, the producer can read another
frame from the database and send it, or just block if there isn't space
for the next message.
In the case of a "live" stream (e.g. video capture card scenario), we
might want some messages to be swapped to disk temporarily, in case we
have a consumer that is running on a machine that is temporarily very
busy. Being able to specify some kind of bound on the size of the Kaha
store would be ideal here. Once this maximum size is reached, producers
could just block on send. Other people might want to let the Kaha store
grow without bound (though this isn't applicable in our case).
In the case where one is sending messages read from database/disk, you
don't want messages being swapped to disk (i.e. maximum size of Kaha
store is set to 0), because a bunch of fast producers can quickly end
up making an extra "copy" of your whole dataset in the broker's Kaha
store (which could get huge). Here, you really just want the producer to
block if there isn't space for more messages.
In the case where the consumers are *REALLY* slow, like none of them
manage to consume any messages inside of N minutes after the producer
has been blocked, it would be nice if a JMSException could be thrown on
the producer side, indicating that a timeout has occured.
Hope this makes sense. Thanks for listening! ;-)
Cheers,
Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by Rob Davies <ra...@gmail.com>.
Hi Albert,
what is the behavior you are expecting ?
cheers,
Rob
On 2 Feb 2007, at 19:23, Albert Strasheim wrote:
> Hello all
>
> On Mon, 29 Jan 2007, Albert Strasheim wrote:
>
>> Hey Rob,
>>
>> Thanks, I'll try this suggestion.
>>
>> I'm seeing a few issues with the new code (producer getting stuck,
>> consumer not getting any messages). I'm working on a test case to
>> show
>> the problem. I'll create a JIRA issue as soon as I have something
>> concrete nailed down.
>
> If anybody is still following along at home, I've created a JIRA issue
> (AMQ-1148) which has a test case attached which shows the problem
> we're
> having with a fast producer/slow consumer system getting stuck after
> having consuming about 10 messages.
>
> Cheers,
>
> Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by Rob Davies <ra...@gmail.com>.
thanks Albert!
On 2 Feb 2007, at 19:23, Albert Strasheim wrote:
> Hello all
>
> On Mon, 29 Jan 2007, Albert Strasheim wrote:
>
>> Hey Rob,
>>
>> Thanks, I'll try this suggestion.
>>
>> I'm seeing a few issues with the new code (producer getting stuck,
>> consumer not getting any messages). I'm working on a test case to
>> show
>> the problem. I'll create a JIRA issue as soon as I have something
>> concrete nailed down.
>
> If anybody is still following along at home, I've created a JIRA issue
> (AMQ-1148) which has a test case attached which shows the problem
> we're
> having with a fast producer/slow consumer system getting stuck after
> having consuming about 10 messages.
>
> Cheers,
>
> Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by Albert Strasheim <13...@sun.ac.za>.
Hello all
On Mon, 29 Jan 2007, Albert Strasheim wrote:
> Hey Rob,
>
> Thanks, I'll try this suggestion.
>
> I'm seeing a few issues with the new code (producer getting stuck,
> consumer not getting any messages). I'm working on a test case to show
> the problem. I'll create a JIRA issue as soon as I have something
> concrete nailed down.
If anybody is still following along at home, I've created a JIRA issue
(AMQ-1148) which has a test case attached which shows the problem we're
having with a fast producer/slow consumer system getting stuck after
having consuming about 10 messages.
Cheers,
Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by Albert Strasheim <13...@sun.ac.za>.
Hey Rob,
Thanks, I'll try this suggestion.
I'm seeing a few issues with the new code (producer getting stuck,
consumer not getting any messages). I'm working on a test case to show
the problem. I'll create a JIRA issue as soon as I have something
concrete nailed down.
Cheers,
Albert
On Mon, 29 Jan 2007, Rob Davies wrote:
> Hi Albert,
>
> there's a few different ways to do this - my preferred way is create
> the broker separately in the same VM - e.g.
>
> BrokerFactoryBean factory = new BrokerFactoryBean(new
> ClassPathResource("org/apache/activemq/broker/policy/cursor.xml"));
> factory.afterPropertiesSet();
> BrokerService broker = factory.getBroker();
> broker.start();
>
> ....
Re: Fast producer, slow consumer with spooling to disk?
Posted by Rob Davies <ra...@gmail.com>.
Hi Albert,
there's a few different ways to do this - my preferred way is create
the broker separately in the same VM - e.g.
BrokerFactoryBean factory = new BrokerFactoryBean(new
ClassPathResource("org/apache/activemq/broker/policy/cursor.xml"));
factory.afterPropertiesSet();
BrokerService broker = factory.getBroker();
broker.start();
....
but ensure the broker has well known name - e.g. fred, then connect
to it using the URI vm://fred.
If the ActiveMQConnectionFactory discovers a broker in the same jvm
with the name you look for, it won't create a new one. By default, a
broker's name is localhost
cheers,
Rob
On 29 Jan 2007, at 12:11, Albert Strasheim wrote:
> Hey Rob
>
> Thanks very much for this. I'd like to try this in our unit tests,
> which
> is the first place we've been running into these problems (testing a
> slow consumer with "fake" data, which is produced very fast).
>
> Is there a way to configure policies using the broker URL or in the
> queue/topic name (after the ?) from code? For our unit tests, we're
> currently using an embedded broker, so this would makes things
> relatively simple. Alternatively, is there a way to make an embedded
> broker parse XML configuration somehow?
>
> Thanks again.
>
> Cheers,
>
> Albert
>
> On Mon, 29 Jan 2007, Rob Davies wrote:
>
>> Hi Albert,
>>
>> from the latest in SVN it's possible to configure the cursor used for
>> a topic subscriber - which by default is file based. If you use a VM
>> based cursor, as messages aren't removed from memory (to spool to
>> disk) - the producer should block. You can configure the cursor type
>> using destination policies: http://incubator.apache.org/activemq/
>> xbean-xml-reference-41.html#XBeanXMLReference4.1-policyEntryelement
>>
>>
>> An example is:
Re: Fast producer, slow consumer with spooling to disk?
Posted by Albert Strasheim <13...@sun.ac.za>.
Hey Rob
Thanks very much for this. I'd like to try this in our unit tests, which
is the first place we've been running into these problems (testing a
slow consumer with "fake" data, which is produced very fast).
Is there a way to configure policies using the broker URL or in the
queue/topic name (after the ?) from code? For our unit tests, we're
currently using an embedded broker, so this would makes things
relatively simple. Alternatively, is there a way to make an embedded
broker parse XML configuration somehow?
Thanks again.
Cheers,
Albert
On Mon, 29 Jan 2007, Rob Davies wrote:
> Hi Albert,
>
> from the latest in SVN it's possible to configure the cursor used for
> a topic subscriber - which by default is file based. If you use a VM
> based cursor, as messages aren't removed from memory (to spool to
> disk) - the producer should block. You can configure the cursor type
> using destination policies: http://incubator.apache.org/activemq/
> xbean-xml-reference-41.html#XBeanXMLReference4.1-policyEntryelement
>
>
> An example is:
Re: Fast producer, slow consumer with spooling to disk?
Posted by Albert Strasheim <fu...@gmail.com>.
Hey Rob
Thanks very much for this. I'd like to try this in our unit tests, which
is the first place we've been running into these problems (testing a
slow consumer with "fake" data, which is produced very fast).
Is there a way to configure policies using the broker URL or in the
queue/topic name (after the ?) from code? For our unit tests, we're
currently using an embedded broker, so this would makes things
relatively simple. Alternatively, is there a way to make an embedded
broker parse XML configuration somehow?
Thanks again.
Cheers,
Albert
On Mon, 29 Jan 2007, Rob Davies wrote:
> Hi Albert,
>
> from the latest in SVN it's possible to configure the cursor used for
> a topic subscriber - which by default is file based. If you use a VM
> based cursor, as messages aren't removed from memory (to spool to
> disk) - the producer should block. You can configure the cursor type
> using destination policies: http://incubator.apache.org/activemq/
> xbean-xml-reference-41.html#XBeanXMLReference4.1-policyEntryelement
>
>
> An example is:
Re: Fast producer, slow consumer with spooling to disk?
Posted by Rob Davies <ra...@gmail.com>.
Hi Albert,
from the latest in SVN it's possible to configure the cursor used for
a topic subscriber - which by default is file based. If you use a VM
based cursor, as messages aren't removed from memory (to spool to
disk) - the producer should block. You can configure the cursor type
using destination policies: http://incubator.apache.org/activemq/
xbean-xml-reference-41.html#XBeanXMLReference4.1-policyEntryelement
An example is:
<beans>
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfi
gurer"/>
<broker persistent="false" xmlns="http://activemq.org/config/1.0">
<!-- lets define the dispatch policy -->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic="org.apache.>">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
<deadLetterStrategy>
<individualDeadLetterStrategy topicPrefix="Test.DLQ." />
</deadLetterStrategy>
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue="org.apache.>">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="Test.DLQ."/>
</deadLetterStrategy>
<pendingQueuePolicy>
<vmQueueCursor />
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
</broker>
</beans>
where the all topic subscribers to destinations starting with
org.apache will have a VMCursor for their pending messages.
hope that helps
cheers,
Rob
On 26 Jan 2007, at 17:53, Albert Strasheim wrote:
> Hey Rob,
>
> On Fri, 26 Jan 2007, Rob Davies wrote:
>
>> Hi Albert,
>> I think you could need to set the prefetch limit as well - it could
>> be the clients running out of memory.
>
> Thanks, I'll experiment with the prefetch limit a bit.
>
>> I agree we need to allow for configuration of blocking a producer
>> whilst waiting for space - be that disk or memory - this may take a
>> few more days to get right :(
>
> Will you have time to look into this? Could you provide some hints
> as to
> where one should start looking to implement this feature.
>
> Thanks for the quick reply and the patches so far!
>
> Cheers,
>
> Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by Albert Strasheim <13...@sun.ac.za>.
Hey Rob,
On Fri, 26 Jan 2007, Rob Davies wrote:
> Hi Albert,
> I think you could need to set the prefetch limit as well - it could
> be the clients running out of memory.
Thanks, I'll experiment with the prefetch limit a bit.
> I agree we need to allow for configuration of blocking a producer
> whilst waiting for space - be that disk or memory - this may take a
> few more days to get right :(
Will you have time to look into this? Could you provide some hints as to
where one should start looking to implement this feature.
Thanks for the quick reply and the patches so far!
Cheers,
Albert
Re: Fast producer, slow consumer with spooling to disk?
Posted by Rob Davies <ra...@gmail.com>.
Hi Albert,
I think you could need to set the prefetch limit as well - it could
be the clients running out of memory.
I agree we need to allow for configuration of blocking a producer
whilst waiting for space - be that disk or memory - this may take a
few more days to get right :(
cheers,
Rob
On 26 Jan 2007, at 16:44, Albert Strasheim wrote:
> Hello all
>
> Following on from these threads:
>
> http://www.nabble.com/Blocking-on-UsageManager.waitForSpace-again-
> tf3031460.html
> http://www.nabble.com/OutOfMemoryErrors-again-tf3083798.html
>
> I've updated my ActiveMQ 4.2 from trunk to include Rob's latest
> patches, but I'm still having problems.
>
> Looking at the example I posted here:
>
> http://www.nabble.com/Re%3A-Blocking-on-UsageManager.waitForSpace-
> again-p8521205.html
>
> is there a way in a fast producer, slow consumer scenario to configure
> ActiveMQ to block the producer until space becomes available for it to
> send more messages, without spooling to disk or running out of memory
> (maybe with some kind of configurable timeout on how long the producer
> blocks before throwing)? We want the messages to move at the speed of
> the slowest consumer (since we only ever have one consumer), not the
> producer.
>
> Running that example, with the Sun JDK's default heap size, I still
> run into OutOfMemoryErrors. Increasing the heap size with -Xms256m
> -Xmx512m causes the program to run, until it promptly fills up the
> disk on my machine. I've specified both broker.persistent=false and
> DeliveryMode.NON_PERSISTENT on the producer, but it seems some kind of
> persistence is still active, since I promptly see my activemq-data
> directory filling up.
>
> Thanks!
>
> Cheers,
>
> Albert