You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2013/04/11 21:57:56 UTC
Re: Data in File-channel data folder
The File Channel deletes the files from which all events have been removed at the time of a checkpoint. But, the channel will keep 2 log files per data directory even if all its events are taken out. Once all events from log-1, log-2 are taken out and then events get written to log-3, log-1 will be deleted at the next checkpoint. Unless a log-3 is created, both log-1 and log-2 will not be deleted.
Hari
--
Hari Shreedharan
On Thursday, April 11, 2013 at 12:13 PM, Madhu Gmail wrote:
> Hello,
>
> I have not heard from anyone. so just want make sure I have explained the issue correctly.
>
> I think this is a common problem for everyone who uses it flume.
>
> when flume sink consumes the log event from file channel, what will happen to the data that is committed to local disk under data directory.
>
> will it grow indefinitely like log-1, log-2, log-3.....and so on ???
>
> do I have to write script to remove the data from data directory ??
>
>
>
> Madhu Munagala
> (214)679-2872
>
> On Apr 11, 2013, at 11:52 AM, Madhu Gmail <madhu.munagala@gmail.com (mailto:madhu.munagala@gmail.com)> wrote:
>
> > Hello,
> >
> > How to clean up the data in file channel data folder. After the log events are processed by the sink, I still see the log-1 and log-2 shows 1.6GB and 1.2GB.
> >
> > once the log events are processed by the sink, the channel should not have any data in data directory under file-channel ....??
> >
> >
> > Madhu Munagala
> > (214)679-2872
> >
>
>
>
Re: Data in File-channel data folder
Posted by Madhu Gmail <ma...@gmail.com>.
Thank you Hari.
This is very useful information.
Let me try that scenario where I will send more data to create log-3.
Thanks
Madhu Munagala
On Apr 11, 2013, at 2:57 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
> The File Channel deletes the files from which all events have been removed at the time of a checkpoint. But, the channel will keep 2 log files per data directory even if all its events are taken out. Once all events from log-1, log-2 are taken out and then events get written to log-3, log-1 will be deleted at the next checkpoint. Unless a log-3 is created, both log-1 and log-2 will not be deleted.
>
>
>
> Hari
>
> --
> Hari Shreedharan
>
> On Thursday, April 11, 2013 at 12:13 PM, Madhu Gmail wrote:
>
>> Hello,
>>
>> I have not heard from anyone. so just want make sure I have explained the issue correctly.
>>
>> I think this is a common problem for everyone who uses it flume.
>>
>> when flume sink consumes the log event from file channel, what will happen to the data that is committed to local disk under data directory.
>>
>> will it grow indefinitely like log-1, log-2, log-3.....and so on ???
>>
>> do I have to write script to remove the data from data directory ??
>>
>>
>>
>> Madhu Munagala
>> (214)679-2872
>>
>> On Apr 11, 2013, at 11:52 AM, Madhu Gmail <ma...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> How to clean up the data in file channel data folder. After the log events are processed by the sink, I still see the log-1 and log-2 shows 1.6GB and 1.2GB.
>>>
>>> once the log events are processed by the sink, the channel should not have any data in data directory under file-channel ....??
>>>
>>>
>>> Madhu Munagala
>>> (214)679-2872
>