Re: Fast producer, slow consumer with spooling to disk?

[Albert Strasheim <13...@sun.ac.za>]

Hello all

On Mon, 29 Jan 2007, Albert Strasheim wrote:

> Hey Rob,
> 
> Thanks, I'll try this suggestion.
> 
> I'm seeing a few issues with the new code (producer getting stuck, 
> consumer not getting any messages). I'm working on a test case to show 
> the problem. I'll create a JIRA issue as soon as I have something 
> concrete nailed down.

If anybody is still following along at home, I've created a JIRA issue 
(AMQ-1148) which has a test case attached which shows the problem we're 
having with a fast producer/slow consumer system getting stuck after 
having consuming about 10 messages.

Cheers,

Albert

Re: Fast producer, slow consumer with spooling to disk?

[Albert Strasheim <13...@sun.ac.za>]

Hello all,

On Fri, 02 Feb 2007, Rob Davies wrote:

> Hi Albert,
> 
> what is the behavior you are expecting ?
> 
> cheers,
> 
> Rob

Hey Rob. What we'd like to happen in our fast producer/slow consumer 
scenario when the broker runs out of space (in memory) is that it 
blocks the producers for a while, letting the slow consumers consume 
some messages to free up space for the producers to send more.

We have this scenario in cases where our producer is something that 
streams large amounts of data from, e.g., disk/database/video capture 
card to a consumer that takes long to process each message (e.g., 
running a bunch of image processing algorithms).

In the case where one streams from something like a video capture card, 
you'll want a policy that starts dropping frames after they haven't been 
consumed for a while. In the database case (say you recorded some video 
on another machine), you don't really want to drop these frames -- as 
long as the consumer keeps ticking over, the producer can read another 
frame from the database and send it, or just block if there isn't space 
for the next message.

In the case of a "live" stream (e.g. video capture card scenario), we 
might want some messages to be swapped to disk temporarily, in case we 
have a consumer that is running on a machine that is temporarily very 
busy. Being able to specify some kind of bound on the size of the Kaha 
store would be ideal here. Once this maximum size is reached, producers 
could just block on send. Other people might want to let the Kaha store 
grow without bound (though this isn't applicable in our case).

In the case where one is sending messages read from database/disk, you 
don't want messages being swapped to disk (i.e. maximum size of Kaha 
store is set to 0),  because a bunch of fast producers can quickly end 
up making an extra "copy" of your whole dataset in the broker's Kaha 
store (which could get huge). Here, you really just want the producer to 
block if there isn't space for more messages.

In the case where the consumers are *REALLY* slow, like none of them 
manage to consume any messages inside of N minutes after the producer 
has been blocked, it would be nice if a JMSException could be thrown on 
the producer side, indicating that a timeout has occured.

Hope this makes sense. Thanks for listening! ;-)

Cheers,

Albert

Re: Fast producer, slow consumer with spooling to disk?

[Rob Davies <ra...@gmail.com>]

Hi Albert,

what is the behavior you are expecting ?

cheers,

Rob

On 2 Feb 2007, at 19:23, Albert Strasheim wrote:

> Hello all
>
> On Mon, 29 Jan 2007, Albert Strasheim wrote:
>
>> Hey Rob,
>>
>> Thanks, I'll try this suggestion.
>>
>> I'm seeing a few issues with the new code (producer getting stuck,
>> consumer not getting any messages). I'm working on a test case to  
>> show
>> the problem. I'll create a JIRA issue as soon as I have something
>> concrete nailed down.
>
> If anybody is still following along at home, I've created a JIRA issue
> (AMQ-1148) which has a test case attached which shows the problem  
> we're
> having with a fast producer/slow consumer system getting stuck after
> having consuming about 10 messages.
>
> Cheers,
>
> Albert

Re: Fast producer, slow consumer with spooling to disk?

[Rob Davies <ra...@gmail.com>]

thanks Albert!

On 2 Feb 2007, at 19:23, Albert Strasheim wrote:

> Hello all
>
> On Mon, 29 Jan 2007, Albert Strasheim wrote:
>
>> Hey Rob,
>>
>> Thanks, I'll try this suggestion.
>>
>> I'm seeing a few issues with the new code (producer getting stuck,
>> consumer not getting any messages). I'm working on a test case to  
>> show
>> the problem. I'll create a JIRA issue as soon as I have something
>> concrete nailed down.
>
> If anybody is still following along at home, I've created a JIRA issue
> (AMQ-1148) which has a test case attached which shows the problem  
> we're
> having with a fast producer/slow consumer system getting stuck after
> having consuming about 10 messages.
>
> Cheers,
>
> Albert