You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Robert Slifka <ro...@gmail.com> on 2012/11/29 01:33:26 UTC
"Standard" HTTP logging stack? (0.9.3 => 1.2 Upgrade)
Hi all,
Simple HTTP access.log aggregation case here. Front ends running Apache
needing to aggregate/sink all access.log data to S3.
Right now we're on OG 0.9.3:
nginx => access.log => tailSource => Collector => S3
For 1.2, I was under the impression we would simplify and move to:
Apache => Syslog => Agent…Agent => S3.
However it looks like Apache can only Syslog the error.log. Am I missing
something?
What's the recommended 'standard' setup?
Thanks guys,
Rob
Re: "Standard" HTTP logging stack? (0.9.3 => 1.2 Upgrade)
Posted by Brock Noland <br...@cloudera.com>.
It's possible you could use the exec source and tail -F but personally I
would use either the Flume 1.3 RC (until it's released).
On Thu, Nov 29, 2012 at 11:52 AM, Robert Slifka <ro...@gmail.com>wrote:
> Hi Brock,
>
> Thanks for the heads up, wasn't aware that was incoming.
>
> If you don't mind me asking, what's the current recommended approach with
> 1.2?
>
> Rob
>
>
> On Thu, Nov 29, 2012 at 9:19 AM, Brock Noland <br...@cloudera.com> wrote:
>
>> I would use the Spool Directory source in 1.3 to upload the logs files
>> once apache rolls the log.
>>
>>
>> On Wed, Nov 28, 2012 at 6:33 PM, Robert Slifka <ro...@gmail.com>wrote:
>>
>>> Hi all,
>>>
>>> Simple HTTP access.log aggregation case here. Front ends running Apache
>>> needing to aggregate/sink all access.log data to S3.
>>>
>>> Right now we're on OG 0.9.3:
>>>
>>> nginx => access.log => tailSource => Collector => S3
>>>
>>>
>>> For 1.2, I was under the impression we would simplify and move to:
>>>
>>> Apache => Syslog => Agent…Agent => S3.
>>>
>>>
>>> However it looks like Apache can only Syslog the error.log. Am I
>>> missing something?
>>>
>>> What's the recommended 'standard' setup?
>>>
>>> Thanks guys,
>>>
>>> Rob
>>>
>>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: "Standard" HTTP logging stack? (0.9.3 => 1.2 Upgrade)
Posted by Robert Slifka <ro...@gmail.com>.
Hi Brock,
Thanks for the heads up, wasn't aware that was incoming.
If you don't mind me asking, what's the current recommended approach with
1.2?
Rob
On Thu, Nov 29, 2012 at 9:19 AM, Brock Noland <br...@cloudera.com> wrote:
> I would use the Spool Directory source in 1.3 to upload the logs files
> once apache rolls the log.
>
>
> On Wed, Nov 28, 2012 at 6:33 PM, Robert Slifka <ro...@gmail.com>wrote:
>
>> Hi all,
>>
>> Simple HTTP access.log aggregation case here. Front ends running Apache
>> needing to aggregate/sink all access.log data to S3.
>>
>> Right now we're on OG 0.9.3:
>>
>> nginx => access.log => tailSource => Collector => S3
>>
>>
>> For 1.2, I was under the impression we would simplify and move to:
>>
>> Apache => Syslog => Agent…Agent => S3.
>>
>>
>> However it looks like Apache can only Syslog the error.log. Am I missing
>> something?
>>
>> What's the recommended 'standard' setup?
>>
>> Thanks guys,
>>
>> Rob
>>
>>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Re: "Standard" HTTP logging stack? (0.9.3 => 1.2 Upgrade)
Posted by Brock Noland <br...@cloudera.com>.
I would use the Spool Directory source in 1.3 to upload the logs files once
apache rolls the log.
On Wed, Nov 28, 2012 at 6:33 PM, Robert Slifka <ro...@gmail.com>wrote:
> Hi all,
>
> Simple HTTP access.log aggregation case here. Front ends running Apache
> needing to aggregate/sink all access.log data to S3.
>
> Right now we're on OG 0.9.3:
>
> nginx => access.log => tailSource => Collector => S3
>
>
> For 1.2, I was under the impression we would simplify and move to:
>
> Apache => Syslog => Agent…Agent => S3.
>
>
> However it looks like Apache can only Syslog the error.log. Am I missing
> something?
>
> What's the recommended 'standard' setup?
>
> Thanks guys,
>
> Rob
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/