You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by terreyshih <te...@gmail.com> on 2014/10/31 03:46:15 UTC
sink buffer event processing order
Hi,
I would like to know what happens in the in following scenario:
1. down stream source agent is down for some reason.
2. up stream agent buffers it in the sink channel while keeps receiving data from the source
3. down stream agent comes back up.
4. does the upstream agent CLEAR the buffer before sending the new incoming data from the source ?
5. Is it possible that the buffer data and the newly incoming data INTERLEAVE each other ???
In other words,
In the upstream agent:
The buffer has 1,2,3,4,5
The new data comes 6,7,8
The upstream agent sends the data in the order of 6,7,1,2,3,8,5, etc.
Again, I know this would not happen if the down stream agent is never brought down. However, I am just wondering if it is possible for this to happen is the downstream is brought down and then up again ?
thanks,
-Gary
Re: sink buffer event processing order
Posted by terreyshih <te...@gmail.com>.
Thanks for the response.
What we are experiencing is that the ordering gets messed up for a while and then after some time, the ordering restores.
This seems to suggest that once the buffering clears, the ordering restores, and that is why I was looking for some help on understanding this.
The problem is that I have got the source avro sink, agent one source/sink and agent two source/sink. We only found out the ordering is messed up at the agent sink when it’s hooked up to POJO business entity.
In order to see where the ordering gets messed up, I would need to put a transaction id/timestamp at the avro source and traces the transaction all the way through the topology to see where the ordering actually gets out of order.
Can you suggest some other way to debug this ?
thx
> On Oct 30, 2014, at 9:39 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>
> Ordering can get messed up if there are multiple sinks removing data and also if there are failures. When multiple sinks remove data, the order depends on the ordering of the sink writes. Same goes for multiple sources.
>
> Thanks,
> Hari
>
>
> On Thu, Oct 30, 2014 at 9:27 PM, Ashish <paliwalashish@gmail.com <ma...@gmail.com>> wrote:
>
> Channel is like a Queue, new data shall be appended at the end. So you
> shall receive data as 1,2,3,4,5,6,7,8 at upstream agent.
>
> On Fri, Oct 31, 2014 at 8:16 AM, terreyshih <te...@gmail.com> wrote:
> > Hi,
> >
> > I would like to know what happens in the in following scenario:
> >
> > 1. down stream source agent is down for some reason.
> > 2. up stream agent buffers it in the sink channel while keeps receiving data from the source
> > 3. down stream agent comes back up.
> > 4. does the upstream agent CLEAR the buffer before sending the new incoming data from the source ?
> > 5. Is it possible that the buffer data and the newly incoming data INTERLEAVE each other ???
> >
> > In other words,
> >
> > In the upstream agent:
> >
> > The buffer has 1,2,3,4,5
> > The new data comes 6,7,8
> >
> > The upstream agent sends the data in the order of 6,7,1,2,3,8,5, etc.
> >
> > Again, I know this would not happen if the down stream agent is never brought down. However, I am just wondering if it is possible for this to happen is the downstream is brought down and then up again ?
> >
> > thanks,
> > -Gary
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
>
Re: sink buffer event processing order
Posted by Hari Shreedharan <hs...@cloudera.com>.
Ordering can get messed up if there are multiple sinks removing data and also if there are failures. When multiple sinks remove data, the order depends on the ordering of the sink writes. Same goes for multiple sources.
Thanks,
Hari
On Thu, Oct 30, 2014 at 9:27 PM, Ashish <pa...@gmail.com> wrote:
> Channel is like a Queue, new data shall be appended at the end. So you
> shall receive data as 1,2,3,4,5,6,7,8 at upstream agent.
> On Fri, Oct 31, 2014 at 8:16 AM, terreyshih <te...@gmail.com> wrote:
>> Hi,
>>
>> I would like to know what happens in the in following scenario:
>>
>> 1. down stream source agent is down for some reason.
>> 2. up stream agent buffers it in the sink channel while keeps receiving data from the source
>> 3. down stream agent comes back up.
>> 4. does the upstream agent CLEAR the buffer before sending the new incoming data from the source ?
>> 5. Is it possible that the buffer data and the newly incoming data INTERLEAVE each other ???
>>
>> In other words,
>>
>> In the upstream agent:
>>
>> The buffer has 1,2,3,4,5
>> The new data comes 6,7,8
>>
>> The upstream agent sends the data in the order of 6,7,1,2,3,8,5, etc.
>>
>> Again, I know this would not happen if the down stream agent is never brought down. However, I am just wondering if it is possible for this to happen is the downstream is brought down and then up again ?
>>
>> thanks,
>> -Gary
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: sink buffer event processing order
Posted by Ashish <pa...@gmail.com>.
Channel is like a Queue, new data shall be appended at the end. So you
shall receive data as 1,2,3,4,5,6,7,8 at upstream agent.
On Fri, Oct 31, 2014 at 8:16 AM, terreyshih <te...@gmail.com> wrote:
> Hi,
>
> I would like to know what happens in the in following scenario:
>
> 1. down stream source agent is down for some reason.
> 2. up stream agent buffers it in the sink channel while keeps receiving data from the source
> 3. down stream agent comes back up.
> 4. does the upstream agent CLEAR the buffer before sending the new incoming data from the source ?
> 5. Is it possible that the buffer data and the newly incoming data INTERLEAVE each other ???
>
> In other words,
>
> In the upstream agent:
>
> The buffer has 1,2,3,4,5
> The new data comes 6,7,8
>
> The upstream agent sends the data in the order of 6,7,1,2,3,8,5, etc.
>
> Again, I know this would not happen if the down stream agent is never brought down. However, I am just wondering if it is possible for this to happen is the downstream is brought down and then up again ?
>
> thanks,
> -Gary
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal