You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Cameron Gandevia <cg...@gmail.com> on 2012/10/13 03:21:16 UTC
Multi-line Events
Hi
We are currently using the original flume and are considering switching to
flume ng using the exec "tail -F" source and eventually the spool directory
source
https://issues.apache.org/jira/browse/FLUME-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
.
My only question is I don't see how to handle multi-line log entries such
as stack traces using either of these methods. Is this possible?
Thanks
Re: Multi-line Events
Posted by Cameron Gandevia <cg...@gmail.com>.
Thanks for the reply, my plan was to write a simple python script to handle
this if there wasn't already a mechanism.
I am also going to look at modifying FLUME-1425 to support pluggable
readers but couldn't get the patch to apply to trunk. I will take a look
again later.
On Oct 13, 2012 7:27 AM, "Brock Noland" <br...@cloudera.com> wrote:
> Hi,
>
> I don't think there is anyway handle multi-line events with exec and
> tail -F source. One could be written in python/perl/etc fairly easily
> but would have to be written.
>
> Brock
>
> On Fri, Oct 12, 2012 at 9:44 PM, Cameron Gandevia <cg...@gmail.com>
> wrote:
> > The plan is to eventually move to using the log4j appender but for now we
> > don't want to introduce any dependencies in our application.
> >
> > On Oct 12, 2012 6:30 PM, "Brock Noland" <br...@cloudera.com> wrote:
> >>
> >> Hi,
> >>
> >> As opposed to using tail, why not use the API or log4j to send events
> >> directly to a flume agent?
> >>
> >> Brock
> >>
> >> On Friday, October 12, 2012 at 8:21 PM, Cameron Gandevia wrote:
> >>
> >> Hi
> >>
> >> We are currently using the original flume and are considering switching
> to
> >> flume ng using the exec "tail -F" source and eventually the spool
> directory
> >> source
> >>
> https://issues.apache.org/jira/browse/FLUME-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> .
> >> My only question is I don't see how to handle multi-line log entries
> such
> >> as stack traces using either of these methods. Is this possible?
> >>
> >> Thanks
> >>
> >>
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
Re: Multi-line Events
Posted by Brock Noland <br...@cloudera.com>.
Hi,
I don't think there is anyway handle multi-line events with exec and
tail -F source. One could be written in python/perl/etc fairly easily
but would have to be written.
Brock
On Fri, Oct 12, 2012 at 9:44 PM, Cameron Gandevia <cg...@gmail.com> wrote:
> The plan is to eventually move to using the log4j appender but for now we
> don't want to introduce any dependencies in our application.
>
> On Oct 12, 2012 6:30 PM, "Brock Noland" <br...@cloudera.com> wrote:
>>
>> Hi,
>>
>> As opposed to using tail, why not use the API or log4j to send events
>> directly to a flume agent?
>>
>> Brock
>>
>> On Friday, October 12, 2012 at 8:21 PM, Cameron Gandevia wrote:
>>
>> Hi
>>
>> We are currently using the original flume and are considering switching to
>> flume ng using the exec "tail -F" source and eventually the spool directory
>> source
>> https://issues.apache.org/jira/browse/FLUME-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel.
>> My only question is I don't see how to handle multi-line log entries such
>> as stack traces using either of these methods. Is this possible?
>>
>> Thanks
>>
>>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Multi-line Events
Posted by Cameron Gandevia <cg...@gmail.com>.
The plan is to eventually move to using the log4j appender but for now we
don't want to introduce any dependencies in our application.
On Oct 12, 2012 6:30 PM, "Brock Noland" <br...@cloudera.com> wrote:
> Hi,
>
> As opposed to using tail, why not use the API or log4j to send events
> directly to a flume agent?
>
> Brock
>
> On Friday, October 12, 2012 at 8:21 PM, Cameron Gandevia wrote:
>
> Hi
>
> We are currently using the original flume and are considering switching to
> flume ng using the exec "tail -F" source and eventually the spool directory
> source
> https://issues.apache.org/jira/browse/FLUME-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> .
> My only question is I don't see how to handle multi-line log entries such
> as stack traces using either of these methods. Is this possible?
>
> Thanks
>
>
>
Re: Multi-line Events
Posted by Brock Noland <br...@cloudera.com>.
Hi,
As opposed to using tail, why not use the API or log4j to send events directly to a flume agent?
Brock
On Friday, October 12, 2012 at 8:21 PM, Cameron Gandevia wrote:
> Hi
>
> We are currently using the original flume and are considering switching to flume ng using the exec "tail -F" source and eventually the spool directory source https://issues.apache.org/jira/browse/FLUME-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel.
> My only question is I don't see how to handle multi-line log entries such as stack traces using either of these methods. Is this possible?
>
> Thanks