You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Jimmy <ji...@gmail.com> on 2014/03/18 01:38:34 UTC
JMS source and full file channel
I have JMS source and file channel with HDFS sink... so far works good
lets assume I hit channel capacity (for whatever reason e.g. HDFS down) and
channel stops accepting new messages from source....
JMS is going to pull a message from queue and will try to delivery to
channel, but cannot, it will rollback. Again, works...
Now if the channel is full for a while, am I going to see JMS pulling
messages from queue like crazy monkey just to roll them back few ms later
or is there some timeout-retry window e.g. "if JMS source cannot delivery 3
messages to channel, wait 30 seconds they retry again"
thank you
Re: JMS source and full file channel
Posted by Jimmy <ji...@gmail.com>.
Thank you, this is very helpful
On Monday, March 17, 2014, Brock Noland <br...@cloudera.com> wrote:
> When the channel is full, the JMSSource returns "BACKOFF" to
> PollableSourceRunner which implements backoff logic which can be seen here:
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java#L139
>
>
> On Mon, Mar 17, 2014 at 7:38 PM, Jimmy <jimmyjack@gmail.com<javascript:_e(%7B%7D,'cvml','jimmyjack@gmail.com');>
> > wrote:
>
>> I have JMS source and file channel with HDFS sink... so far works good
>>
>> lets assume I hit channel capacity (for whatever reason e.g. HDFS down)
>> and channel stops accepting new messages from source....
>>
>> JMS is going to pull a message from queue and will try to delivery to
>> channel, but cannot, it will rollback. Again, works...
>>
>> Now if the channel is full for a while, am I going to see JMS pulling
>> messages from queue like crazy monkey just to roll them back few ms later
>> or is there some timeout-retry window e.g. "if JMS source cannot delivery 3
>> messages to channel, wait 30 seconds they retry again"
>>
>> thank you
>>
>>
>>
>>
>
Re: JMS source and full file channel
Posted by Brock Noland <br...@cloudera.com>.
When the channel is full, the JMSSource returns "BACKOFF" to
PollableSourceRunner which implements backoff logic which can be seen here:
https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/PollableSourceRunner.java#L139
On Mon, Mar 17, 2014 at 7:38 PM, Jimmy <ji...@gmail.com> wrote:
> I have JMS source and file channel with HDFS sink... so far works good
>
> lets assume I hit channel capacity (for whatever reason e.g. HDFS down)
> and channel stops accepting new messages from source....
>
> JMS is going to pull a message from queue and will try to delivery to
> channel, but cannot, it will rollback. Again, works...
>
> Now if the channel is full for a while, am I going to see JMS pulling
> messages from queue like crazy monkey just to roll them back few ms later
> or is there some timeout-retry window e.g. "if JMS source cannot delivery 3
> messages to channel, wait 30 seconds they retry again"
>
> thank you
>
>
>
>