You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Roshan Naik <ro...@hortonworks.com> on 2015/04/02 02:09:21 UTC
File channel and FIFO ordering
Specifically in the case of a single source and sink attached to the channel... I know that in Flume we do not explicitly guarantee FIFO order for any of the channels although in practice the channels do seem to exhibit FIFO.
Specifically wrt to file channel...
Not sure if under normal operation (I.e source and sink are not experiencing issues causing them to abort TXNs) FIFO ordering can be lost.
But...maybe the order can be lost if the single sink is experiencing lot of issues leading to repeatedly aborting TXNs for an extended period of time. Possible ??
Or perhaps there are other known edge cases when FIFO order can be lost ?
-roshan
Re: File channel and FIFO ordering
Posted by Hari Shreedharan <hs...@cloudera.com>.
None which get packaged with Flume.
Thanks, Hari
On Wed, Apr 1, 2015 at 5:24 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> I can't think of any sinks which can do that. Do you know of any such
> sinks ? HDFS sink (I don't think) ? Hbase sink ?
> -roshan
> On 4/1/15 5:17 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>If a sink continuously rolls back, but it is single threaded - only one
>>txn at a time, then the ordering is still guaranteed. But if the sink
>>keeps multiple txns open, then the ordering is gone.
>>
>>
>>
>>Thanks, Hari
>>
>>On Wed, Apr 1, 2015 at 5:10 PM, Roshan Naik <ro...@hortonworks.com>
>>wrote:
>>
>>> Specifically in the case of a single source and sink attached to the
>>>channel... I know that in Flume we do not explicitly guarantee FIFO
>>>order for any of the channels although in practice the channels do seem
>>>to exhibit FIFO.
>>> Specifically wrt to file channel...
>>> Not sure if under normal operation (I.e source and sink are not
>>>experiencing issues causing them to abort TXNs) FIFO ordering can be
>>>lost.
>>> But...maybe the order can be lost if the single sink is experiencing
>>>lot of issues leading to repeatedly aborting TXNs for an extended period
>>>of time. Possible ??
>>> Or perhaps there are other known edge cases when FIFO order can be lost
>>>?
>>> -roshan
Re: File channel and FIFO ordering
Posted by Roshan Naik <ro...@hortonworks.com>.
I can't think of any sinks which can do that. Do you know of any such
sinks ? HDFS sink (I don't think) ? Hbase sink ?
-roshan
On 4/1/15 5:17 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>If a sink continuously rolls back, but it is single threaded - only one
>txn at a time, then the ordering is still guaranteed. But if the sink
>keeps multiple txns open, then the ordering is gone.
>
>
>
>Thanks, Hari
>
>On Wed, Apr 1, 2015 at 5:10 PM, Roshan Naik <ro...@hortonworks.com>
>wrote:
>
>> Specifically in the case of a single source and sink attached to the
>>channel... I know that in Flume we do not explicitly guarantee FIFO
>>order for any of the channels although in practice the channels do seem
>>to exhibit FIFO.
>> Specifically wrt to file channel...
>> Not sure if under normal operation (I.e source and sink are not
>>experiencing issues causing them to abort TXNs) FIFO ordering can be
>>lost.
>> But...maybe the order can be lost if the single sink is experiencing
>>lot of issues leading to repeatedly aborting TXNs for an extended period
>>of time. Possible ??
>> Or perhaps there are other known edge cases when FIFO order can be lost
>>?
>> -roshan
Re: File channel and FIFO ordering
Posted by Hari Shreedharan <hs...@cloudera.com>.
If a sink continuously rolls back, but it is single threaded - only one txn at a time, then the ordering is still guaranteed. But if the sink keeps multiple txns open, then the ordering is gone.
Thanks, Hari
On Wed, Apr 1, 2015 at 5:10 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> Specifically in the case of a single source and sink attached to the channel... I know that in Flume we do not explicitly guarantee FIFO order for any of the channels although in practice the channels do seem to exhibit FIFO.
> Specifically wrt to file channel...
> Not sure if under normal operation (I.e source and sink are not experiencing issues causing them to abort TXNs) FIFO ordering can be lost.
> But...maybe the order can be lost if the single sink is experiencing lot of issues leading to repeatedly aborting TXNs for an extended period of time. Possible ??
> Or perhaps there are other known edge cases when FIFO order can be lost ?
> -roshan