You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Shengjie Min <ke...@gmail.com> on 2012/12/04 18:26:34 UTC
flume write ahead log is finished
Hi,
When it comes to maintenance outage, flume needs to be shutdown and offline
for a while. We want to make sure flume's WAL is completed before we shut
down the process. Is there anyway or any metrics exposed can indicate WAL
is fully processed so that it's safe to bring it down gracefully?
Of coz, the assumption here is that, the source of flume is not writing any
new data, flume only has a certain size backlog needs to be cleaned up.
--
All the best,
Shengjie Min
Re: flume write ahead log is finished
Posted by Shengjie Min <ke...@gmail.com>.
sorry, I should've mentioned this at the beginning, we are running
flume 0.9.4-cdh3u4. It's not 1.3 unfortunately.
Shengjie
On 4 December 2012 21:08, Brock Noland <br...@cloudera.com> wrote:
> You probably want the JSon reporting:
> http://flume.apache.org/FlumeUserGuide.html#monitoring
>
> Brock
>
> On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <ke...@gmail.com> wrote:
> >>Generally you should be fine to bring down the file channel at any
> >>point. That is why we have the WAL.
> >
> > I understand WAL is there in case you restart your flume agent. But in my
> > case I am switching from one flume agent to another type, let's say from
> > tailsource to AMQP agent. So before I shutdown tailsource I really want
> to
> > make sure WAL is all finished. How do you export the channel size in this
> > case?
> >
> >
> > On 4 December 2012 17:36, Brock Noland <br...@cloudera.com> wrote:
> >>
> >> Generally you should be fine to bring down the file channel at any
> >> point. That is why we have the WAL.
> >>
> >> If you wanted to be sure the channel was empty, we do export the
> >> channel size (depth).
> >>
> >> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <ke...@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > When it comes to maintenance outage, flume needs to be shutdown and
> >> > offline
> >> > for a while. We want to make sure flume's WAL is completed before we
> >> > shut
> >> > down the process. Is there anyway or any metrics exposed can indicate
> >> > WAL is
> >> > fully processed so that it's safe to bring it down gracefully?
> >> >
> >> > Of coz, the assumption here is that, the source of flume is not
> writing
> >> > any
> >> > new data, flume only has a certain size backlog needs to be cleaned
> up.
> >> >
> >> >
> >> > --
> >> > All the best,
> >> > Shengjie Min
> >> >
> >>
> >>
> >>
> >> --
> >> Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >
> >
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
All the best,
Shengjie Min
Re: flume write ahead log is finished
Posted by Brock Noland <br...@cloudera.com>.
You probably want the JSon reporting:
http://flume.apache.org/FlumeUserGuide.html#monitoring
Brock
On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <ke...@gmail.com> wrote:
>>Generally you should be fine to bring down the file channel at any
>>point. That is why we have the WAL.
>
> I understand WAL is there in case you restart your flume agent. But in my
> case I am switching from one flume agent to another type, let's say from
> tailsource to AMQP agent. So before I shutdown tailsource I really want to
> make sure WAL is all finished. How do you export the channel size in this
> case?
>
>
> On 4 December 2012 17:36, Brock Noland <br...@cloudera.com> wrote:
>>
>> Generally you should be fine to bring down the file channel at any
>> point. That is why we have the WAL.
>>
>> If you wanted to be sure the channel was empty, we do export the
>> channel size (depth).
>>
>> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <ke...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > When it comes to maintenance outage, flume needs to be shutdown and
>> > offline
>> > for a while. We want to make sure flume's WAL is completed before we
>> > shut
>> > down the process. Is there anyway or any metrics exposed can indicate
>> > WAL is
>> > fully processed so that it's safe to bring it down gracefully?
>> >
>> > Of coz, the assumption here is that, the source of flume is not writing
>> > any
>> > new data, flume only has a certain size backlog needs to be cleaned up.
>> >
>> >
>> > --
>> > All the best,
>> > Shengjie Min
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
>
>
> --
> All the best,
> Shengjie Min
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: flume write ahead log is finished
Posted by Shengjie Min <ke...@gmail.com>.
>Generally you should be fine to bring down the file channel at any
>point. That is why we have the WAL.
I understand WAL is there in case you restart your flume agent. But in my
case I am switching from one flume agent to another type, let's say from
tailsource to AMQP agent. So before I shutdown tailsource I really want to
make sure WAL is all finished. How do you export the channel size in this
case?
On 4 December 2012 17:36, Brock Noland <br...@cloudera.com> wrote:
> Generally you should be fine to bring down the file channel at any
> point. That is why we have the WAL.
>
> If you wanted to be sure the channel was empty, we do export the
> channel size (depth).
>
> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <ke...@gmail.com>
> wrote:
> > Hi,
> >
> > When it comes to maintenance outage, flume needs to be shutdown and
> offline
> > for a while. We want to make sure flume's WAL is completed before we shut
> > down the process. Is there anyway or any metrics exposed can indicate
> WAL is
> > fully processed so that it's safe to bring it down gracefully?
> >
> > Of coz, the assumption here is that, the source of flume is not writing
> any
> > new data, flume only has a certain size backlog needs to be cleaned up.
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
All the best,
Shengjie Min
Re: flume write ahead log is finished
Posted by Brock Noland <br...@cloudera.com>.
Generally you should be fine to bring down the file channel at any
point. That is why we have the WAL.
If you wanted to be sure the channel was empty, we do export the
channel size (depth).
On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <ke...@gmail.com> wrote:
> Hi,
>
> When it comes to maintenance outage, flume needs to be shutdown and offline
> for a while. We want to make sure flume's WAL is completed before we shut
> down the process. Is there anyway or any metrics exposed can indicate WAL is
> fully processed so that it's safe to bring it down gracefully?
>
> Of coz, the assumption here is that, the source of flume is not writing any
> new data, flume only has a certain size backlog needs to be cleaned up.
>
>
> --
> All the best,
> Shengjie Min
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/