You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Shengjie Min <ke...@gmail.com> on 2012/12/05 11:07:03 UTC
How to make sure flume WAL is finished before shut it down
I've asked this question in the flume user group, didn't quite get the
answer.
we are running flume 0.9.4, We want to make sure flume's WAL is completed
before we shut down the flume process. Is there anyway or any metrics
exposed can indicate WAL is fully processed so that it's safe to bring it
down gracefully? How do I know flume has finished all the data?
In our case, we are switching to another input approach, so flume will not
be getting any inputs, it only has a certain size backlog needs to be
cleaned up. So it's more like making sure all the data is delivered before
gracefully shutdown this flume agent forever.
--
All the best,
Shengjie Min
Re: How to make sure flume WAL is finished before shut it down
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
moving back to user@
Hi,
You can monitor the directories (usually /tmp/flume*) for not delivered events, or using the reports from the webUI.
https://cwiki.apache.org/confluence/display/FLUME/Monitoring+Flume
Since FlumeOG (0.9x) will not actively developed anymore, we strongly suggest to move into Flume 1.3.x and up. More informations you'll find in our wikie:
https://cwiki.apache.org/confluence/display/FLUME/Home
Regards,
Alex
On Dec 5, 2012, at 11:07 AM, Shengjie Min <ke...@gmail.com> wrote:
> I've asked this question in the flume user group, didn't quite get the
> answer.
>
> we are running flume 0.9.4, We want to make sure flume's WAL is completed
> before we shut down the flume process. Is there anyway or any metrics
> exposed can indicate WAL is fully processed so that it's safe to bring it
> down gracefully? How do I know flume has finished all the data?
>
> In our case, we are switching to another input approach, so flume will not
> be getting any inputs, it only has a certain size backlog needs to be
> cleaned up. So it's more like making sure all the data is delivered before
> gracefully shutdown this flume agent forever.
>
> --
> All the best,
> Shengjie Min
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: How to make sure flume WAL is finished before shut it down
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
moving back to user@
Hi,
You can monitor the directories (usually /tmp/flume*) for not delivered events, or using the reports from the webUI.
https://cwiki.apache.org/confluence/display/FLUME/Monitoring+Flume
Since FlumeOG (0.9x) will not actively developed anymore, we strongly suggest to move into Flume 1.3.x and up. More informations you'll find in our wikie:
https://cwiki.apache.org/confluence/display/FLUME/Home
Regards,
Alex
On Dec 5, 2012, at 11:07 AM, Shengjie Min <ke...@gmail.com> wrote:
> I've asked this question in the flume user group, didn't quite get the
> answer.
>
> we are running flume 0.9.4, We want to make sure flume's WAL is completed
> before we shut down the flume process. Is there anyway or any metrics
> exposed can indicate WAL is fully processed so that it's safe to bring it
> down gracefully? How do I know flume has finished all the data?
>
> In our case, we are switching to another input approach, so flume will not
> be getting any inputs, it only has a certain size backlog needs to be
> cleaned up. So it's more like making sure all the data is delivered before
> gracefully shutdown this flume agent forever.
>
> --
> All the best,
> Shengjie Min
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: How to make sure flume WAL is finished before shut it down
Posted by Shengjie Min <ke...@gmail.com>.
Is it the case just to go to $flume.agent.logdir, and check if there is any
WAL files left there? if the directory is empty, it means WAL has been
dealt with, then we are safe to shutdown flume process?
On 5 December 2012 10:07, Shengjie Min <ke...@gmail.com> wrote:
> I've asked this question in the flume user group, didn't quite get the
> answer.
>
> we are running flume 0.9.4, We want to make sure flume's WAL is completed
> before we shut down the flume process. Is there anyway or any metrics
> exposed can indicate WAL is fully processed so that it's safe to bring it
> down gracefully? How do I know flume has finished all the data?
>
> In our case, we are switching to another input approach, so flume will not
> be getting any inputs, it only has a certain size backlog needs to be
> cleaned up. So it's more like making sure all the data is delivered before
> gracefully shutdown this flume agent forever.
>
> --
> All the best,
> Shengjie Min
>
>
--
All the best,
Shengjie Min