You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Ryan Chan <ry...@gmail.com> on 2012/02/25 16:39:35 UTC
Centralizing web server access logs - syslog or tailing log file?
I am new to Flume, just a quick question:
Q, I have 10 web servers and need to centralize the access log for analyses
purpose, I have done some research and basically you have two way to send
the log to the "collector", that is using the syslog or tailing the access
log files.
In term of scalability and reliability, which method you would suggest? For
me, seems syslog is more reliable. Any comments?
Thanks.
Re: Centralizing web server access logs - syslog or tailing log file?
Posted by alo alt <wg...@googlemail.com>.
Hi,
Windows is not supported so far. If you have time for patches or testing it would be glad!
best,
Alex
--
Alexander Lorenz
http://mapredit.blogspot.com
On Feb 27, 2012, at 2:36 PM, Chalcy Raja wrote:
> One more question, how about windows flume agent? I do not see any reference to collecting logs in windows.
>
> Thanks,
> Chalcy
>
> From: Chalcy Raja [mailto:Chalcy.Raja@careerbuilder.com]
> Sent: Monday, February 27, 2012 8:31 AM
> To: flume-user@incubator.apache.org
> Subject: RE: Centralizing web server access logs - syslog or tailing log file?
>
> Really?, no master and no zoo keeper?. I like that. I’ll look into getting that to work for us. I’ll let you all know how this is working for us.
>
> Thanks,
> CHalcy
>
> From: Alexander Lorenz [mailto:wget.null@googlemail.com]
> Sent: Sunday, February 26, 2012 4:05 AM
> To: flume-user@incubator.apache.org
> Cc: flume-user@incubator.apache.org
> Subject: Re: Centralizing web server access logs - syslog or tailing log file?
>
> ah, missed that. Yes, no master, no zookeeper. Better design, more flexibility.
>
> sent via my mobile device
>
> On Feb 26, 2012, at 6:35 AM, Ryan Chan <ry...@gmail.com> wrote:
>
> Hi
>
> On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com> wrote:
> Hi,
>
> Did you took an eye to flumeNG? Flume is completely rewritten and match perfect your project.
> https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
>
> best,
> Alex
>
>
>
>
> Thanks for comments, then I will also go for syslog approach.
>
> Btw, since NG is still experimental ATM, or the wiki page is outdated?
>
> I particularly like the simpler design, e.g. no master, no ZK, but does it mean the high availability and centralized control also removed?
>
>
> Thanks
>
RE: Centralizing web server access logs - syslog or tailing log
file?
Posted by Chalcy Raja <Ch...@careerbuilder.com>.
One more question, how about windows flume agent? I do not see any reference to collecting logs in windows.
Thanks,
Chalcy
From: Chalcy Raja [mailto:Chalcy.Raja@careerbuilder.com]
Sent: Monday, February 27, 2012 8:31 AM
To: flume-user@incubator.apache.org
Subject: RE: Centralizing web server access logs - syslog or tailing log file?
Really?, no master and no zoo keeper?. I like that. I’ll look into getting that to work for us. I’ll let you all know how this is working for us.
Thanks,
CHalcy
From: Alexander Lorenz [mailto:wget.null@googlemail.com]
Sent: Sunday, February 26, 2012 4:05 AM
To: flume-user@incubator.apache.org
Cc: flume-user@incubator.apache.org
Subject: Re: Centralizing web server access logs - syslog or tailing log file?
ah, missed that. Yes, no master, no zookeeper. Better design, more flexibility.
sent via my mobile device
On Feb 26, 2012, at 6:35 AM, Ryan Chan <ry...@gmail.com>> wrote:
Hi
On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com>> wrote:
Hi,
Did you took an eye to flumeNG? Flume is completely rewritten and match perfect your project.
https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
best,
Alex
Thanks for comments, then I will also go for syslog approach.
Btw, since NG is still experimental ATM, or the wiki page is outdated?
I particularly like the simpler design, e.g. no master, no ZK, but does it mean the high availability and centralized control also removed?
Thanks
RE: Centralizing web server access logs - syslog or tailing log
file?
Posted by Chalcy Raja <Ch...@careerbuilder.com>.
Really?, no master and no zoo keeper?. I like that. I’ll look into getting that to work for us. I’ll let you all know how this is working for us.
Thanks,
CHalcy
From: Alexander Lorenz [mailto:wget.null@googlemail.com]
Sent: Sunday, February 26, 2012 4:05 AM
To: flume-user@incubator.apache.org
Cc: flume-user@incubator.apache.org
Subject: Re: Centralizing web server access logs - syslog or tailing log file?
ah, missed that. Yes, no master, no zookeeper. Better design, more flexibility.
sent via my mobile device
On Feb 26, 2012, at 6:35 AM, Ryan Chan <ry...@gmail.com>> wrote:
Hi
On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com>> wrote:
Hi,
Did you took an eye to flumeNG? Flume is completely rewritten and match perfect your project.
https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
best,
Alex
Thanks for comments, then I will also go for syslog approach.
Btw, since NG is still experimental ATM, or the wiki page is outdated?
I particularly like the simpler design, e.g. no master, no ZK, but does it mean the high availability and centralized control also removed?
Thanks
Re: Centralizing web server access logs - syslog or tailing log file?
Posted by Alexander Lorenz <wg...@googlemail.com>.
ah, missed that. Yes, no master, no zookeeper. Better design, more flexibility.
sent via my mobile device
On Feb 26, 2012, at 6:35 AM, Ryan Chan <ry...@gmail.com> wrote:
> Hi
>
> On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com> wrote:
> Hi,
>
> Did you took an eye to flumeNG? Flume is completely rewritten and match perfect your project.
> https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
>
> best,
> Alex
>
>
>
>
> Thanks for comments, then I will also go for syslog approach.
>
> Btw, since NG is still experimental ATM, or the wiki page is outdated?
>
> I particularly like the simpler design, e.g. no master, no ZK, but does it mean the high availability and centralized control also removed?
>
>
> Thanks
>
>
Re: Centralizing web server access logs - syslog or tailing log file?
Posted by Ryan Chan <ry...@gmail.com>.
Just have a look today..too bad it does not currently support S3 as the sink
On Sun, Feb 26, 2012 at 5:03 PM, Alexander Lorenz
<wg...@googlemail.com>wrote:
> flumeNG is still alpha / snapshot but usable . we overwork the wiki atm.
>
> best
> Alex
>
> sent via my mobile device
>
> On Feb 26, 2012, at 6:35 AM, Ryan Chan <ry...@gmail.com> wrote:
>
> Hi
>
> On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com> wrote:
>
>> Hi,
>>
>> Did you took an eye to flumeNG? Flume is completely rewritten and match
>> perfect your project.
>>
>> https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
>>
>> best,
>> Alex
>>
>>
>>
>
>
> Thanks for comments, then I will also go for syslog approach.
>
> Btw, since NG is still experimental ATM, or the wiki page is outdated?
>
> I particularly like the simpler design, e.g. no master, no ZK, but does it
> mean the high availability and centralized control also removed?
>
>
> Thanks
>
>
>
Re: Centralizing web server access logs - syslog or tailing log file?
Posted by Alexander Lorenz <wg...@googlemail.com>.
flumeNG is still alpha / snapshot but usable . we overwork the wiki atm.
best
Alex
sent via my mobile device
On Feb 26, 2012, at 6:35 AM, Ryan Chan <ry...@gmail.com> wrote:
> Hi
>
> On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com> wrote:
> Hi,
>
> Did you took an eye to flumeNG? Flume is completely rewritten and match perfect your project.
> https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
>
> best,
> Alex
>
>
>
>
> Thanks for comments, then I will also go for syslog approach.
>
> Btw, since NG is still experimental ATM, or the wiki page is outdated?
>
> I particularly like the simpler design, e.g. no master, no ZK, but does it mean the high availability and centralized control also removed?
>
>
> Thanks
>
>
Re: Centralizing web server access logs - syslog or tailing log file?
Posted by Ryan Chan <ry...@gmail.com>.
Hi
On Sun, Feb 26, 2012 at 2:25 AM, alo alt <wg...@googlemail.com> wrote:
> Hi,
>
> Did you took an eye to flumeNG? Flume is completely rewritten and match
> perfect your project.
>
> https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
>
> best,
> Alex
>
>
>
Thanks for comments, then I will also go for syslog approach.
Btw, since NG is still experimental ATM, or the wiki page is outdated?
I particularly like the simpler design, e.g. no master, no ZK, but does it
mean the high availability and centralized control also removed?
Thanks
Re: Centralizing web server access logs - syslog or tailing log file?
Posted by alo alt <wg...@googlemail.com>.
Hi,
yes, use syslog. tail and tailDir has some limitations and issues.
Did you took an eye to flumeNG? Flume is completely rewritten and match perfect your project.
https://cwiki.apache.org/FLUME/getting-started.html#GettingStarted-BuildingFromSource
best,
Alex
--
Alexander Lorenz
http://mapredit.blogspot.com
On Feb 25, 2012, at 7:39 AM, Ryan Chan wrote:
> I am new to Flume, just a quick question:
>
> Q, I have 10 web servers and need to centralize the access log for analyses purpose, I have done some research and basically you have two way to send the log to the "collector", that is using the syslog or tailing the access log files.
>
> In term of scalability and reliability, which method you would suggest? For me, seems syslog is more reliable. Any comments?
>
> Thanks.