You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Andrew McNair <an...@gmail.com> on 2011/11/10 01:10:11 UTC
using flume to manage all log rotation
Hi,
I'm interested in using flume to manage all log rotation in my deployment.
Currently flume is tailing a logfile. Every night log4j creates a gz backup
of the previous day's logfile. There's a cron job to delete older gz backup
files.
The problems with this approach are:
- Log4j's (apache-log4j-extras's) RollingFileAppender isn't super robust. A
few days ago it locked up all threads while it gzipped the logfiles
- I need a strategy to handle the flume agent going down and missing events
because log4j has rotated the file flume is tailing
- Once Flume has delivered the content of the logfile I don't need the gz
backups, so they're just wasting space
What I'd really like is to have flume handle rolling over the original
files, and deleting backup files once flume is fairly confident it's
delivered the contents to the source. Has anybody setup something like
this? Any pointers would be much appreciated.
Thanks,
Andrew
Re: using flume to manage all log rotation
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Mingjie - thanks for the kind words about search-hadoop.com! :)
Yes, Flume is on the list of projects to add!
Tell us if you are missing any features.
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
>________________________________
>From: Mingjie Lai <mj...@gmail.com>
>To: flume-user@incubator.apache.org
>Sent: Thursday, November 10, 2011 2:27 PM
>Subject: Re: using flume to manage all log rotation
>
>
>+1 logrotate.
>
>Off the topic a little bit:
>
>@Otis. Can sematext add flume project to search-hadoop.com?
>search-hadoop's an awesome tool. So later on, we can point to it for
>previous flume email threads.
>
>Thanks,
>Mingjie
>
>On 11/09/2011 07:46 PM, Otis Gospodnetic wrote:
>> Hi Andrew,
>>
>> We use logrotate to rotate, gzip, and purge old log files.
>> We also use tail -F from Flume because Flume's own tail has its issues
>> (discussed a lot on the old Flume ML 6 or more months ago)
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>> ------------------------------------------------------------------------
>> *From:* Andrew McNair <an...@gmail.com>
>> *To:* flume-user@incubator.apache.org
>> *Sent:* Wednesday, November 9, 2011 7:10 PM
>> *Subject:* using flume to manage all log rotation
>>
>> Hi,
>>
>> I'm interested in using flume to manage all log rotation in my
>> deployment. Currently flume is tailing a logfile. Every night log4j
>> creates a gz backup of the previous day's logfile. There's a cron
>> job to delete older gz backup files.
>>
>> The problems with this approach are:
>> - Log4j's (apache-log4j-extras's) RollingFileAppender isn't super
>> robust. A few days ago it locked up all threads while it gzipped the
>> logfiles
>> - I need a strategy to handle the flume agent going down and missing
>> events because log4j has rotated the file flume is tailing
>> - Once Flume has delivered the content of the logfile I don't need
>> the gz backups, so they're just wasting space
>>
>> What I'd really like is to have flume handle rolling over the
>> original files, and deleting backup files once flume is fairly
>> confident it's delivered the contents to the source. Has anybody
>> setup something like this? Any pointers would be much appreciated.
>>
>> Thanks,
>> Andrew
>>
>>
>
>
>
Re: using flume to manage all log rotation
Posted by Mingjie Lai <mj...@gmail.com>.
+1 logrotate.
Off the topic a little bit:
@Otis. Can sematext add flume project to search-hadoop.com?
search-hadoop's an awesome tool. So later on, we can point to it for
previous flume email threads.
Thanks,
Mingjie
On 11/09/2011 07:46 PM, Otis Gospodnetic wrote:
> Hi Andrew,
>
> We use logrotate to rotate, gzip, and purge old log files.
> We also use tail -F from Flume because Flume's own tail has its issues
> (discussed a lot on the old Flume ML 6 or more months ago)
>
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
> ------------------------------------------------------------------------
> *From:* Andrew McNair <an...@gmail.com>
> *To:* flume-user@incubator.apache.org
> *Sent:* Wednesday, November 9, 2011 7:10 PM
> *Subject:* using flume to manage all log rotation
>
> Hi,
>
> I'm interested in using flume to manage all log rotation in my
> deployment. Currently flume is tailing a logfile. Every night log4j
> creates a gz backup of the previous day's logfile. There's a cron
> job to delete older gz backup files.
>
> The problems with this approach are:
> - Log4j's (apache-log4j-extras's) RollingFileAppender isn't super
> robust. A few days ago it locked up all threads while it gzipped the
> logfiles
> - I need a strategy to handle the flume agent going down and missing
> events because log4j has rotated the file flume is tailing
> - Once Flume has delivered the content of the logfile I don't need
> the gz backups, so they're just wasting space
>
> What I'd really like is to have flume handle rolling over the
> original files, and deleting backup files once flume is fairly
> confident it's delivered the contents to the source. Has anybody
> setup something like this? Any pointers would be much appreciated.
>
> Thanks,
> Andrew
>
>
Re: using flume to manage all log rotation
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Andrew,
We use logrotate to rotate, gzip, and purge old log files.
We also use tail -F from Flume because Flume's own tail has its issues (discussed a lot on the old Flume ML 6 or more months ago)
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/
>________________________________
>From: Andrew McNair <an...@gmail.com>
>To: flume-user@incubator.apache.org
>Sent: Wednesday, November 9, 2011 7:10 PM
>Subject: using flume to manage all log rotation
>
>
>Hi,
>
>I'm interested in using flume to manage all log rotation in my deployment. Currently flume is tailing a logfile. Every night log4j creates a gz backup of the previous day's logfile. There's a cron job to delete older gz backup files.
>
>The problems with this approach are:
>- Log4j's (apache-log4j-extras's) RollingFileAppender isn't super robust. A few days ago it locked up all threads while it gzipped the logfiles
>- I need a strategy to handle the flume agent going down and missing events because log4j has rotated the file flume is tailing
>- Once Flume has delivered the content of the logfile I don't need the gz backups, so they're just wasting space
>
>What I'd really like is to have flume handle rolling over the original files, and deleting backup files once flume is fairly confident it's delivered the contents to the source. Has anybody setup something like this? Any pointers would be much appreciated.
>
>Thanks,
> Andrew
>
>
>