Estimating the event loss

[Buntu Dev <bu...@gmail.com>]

I'm using a Memory channel with capacity set to 100000. Does this mean when
the flume agent restarts, its possible that I loose about 100k events?

For any other durable channel say File channel I noticed .tmp files created
and written to but when I restart the agent these .tmp files are left
as-is. Since some events seem to be written to the .tmp files, should I
expect loss of events in this scenario?


Thanks!

Re: Estimating the event loss

[Hari Shreedharan <hs...@cloudera.com>]

Note that if you simply reconfigure (rather than kill the JVM and restart)
you would not lose data in the memory channel


Thanks,
Hari

On Fri, Jul 17, 2015 at 7:08 AM, Johny Rufus <jr...@cloudera.com> wrote:

> When you restart the agent, all the events in the memory channel will be
> lost (if the memory channel was full and contained 100k events, they will
> be lost)
> File channel will not result in loss of events, on restart the channel's
> state will be restored (File channel always maintains the last two data
> files the events were written to, even if the events in them were
> completely drained, this may give the impression that it still has data
> left)
>
> Thanks,
> Rufus
>
> On Thu, Jul 16, 2015 at 11:10 PM, Buntu Dev <bu...@gmail.com> wrote:
>
>> I'm using a Memory channel with capacity set to 100000. Does this mean
>> when the flume agent restarts, its possible that I loose about 100k events?
>>
>> For any other durable channel say File channel I noticed .tmp files
>> created and written to but when I restart the agent these .tmp files are
>> left as-is. Since some events seem to be written to the .tmp files, should
>> I expect loss of events in this scenario?
>>
>>
>> Thanks!
>>
>
>

Re: Estimating the event loss

[Johny Rufus <jr...@cloudera.com>]

When you restart the agent, all the events in the memory channel will be
lost (if the memory channel was full and contained 100k events, they will
be lost)
File channel will not result in loss of events, on restart the channel's
state will be restored (File channel always maintains the last two data
files the events were written to, even if the events in them were
completely drained, this may give the impression that it still has data
left)

Thanks,
Rufus

On Thu, Jul 16, 2015 at 11:10 PM, Buntu Dev <bu...@gmail.com> wrote:

> I'm using a Memory channel with capacity set to 100000. Does this mean
> when the flume agent restarts, its possible that I loose about 100k events?
>
> For any other durable channel say File channel I noticed .tmp files
> created and written to but when I restart the agent these .tmp files are
> left as-is. Since some events seem to be written to the .tmp files, should
> I expect loss of events in this scenario?
>
>
> Thanks!
>