You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Robbie Strickland (JIRA)" <ji...@apache.org> on 2013/05/09 17:25:16 UTC
[jira] [Updated] (FLUME-2041) Regex interceptor does not populate
headers when using Log4j 2 FlumeAppender
[ https://issues.apache.org/jira/browse/FLUME-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robbie Strickland updated FLUME-2041:
-------------------------------------
Description:
When using a source configured as follows:
{quote}
alerts.sources.a1.type = avro
alerts.sources.a1.channels = c1
alerts.sources.a1.bind = 0.0.0.0
alerts.sources.a1.port = 4141
alerts.sources.a1.interceptors = i1
alerts.sources.a1.interceptors.i1.type = regex_extractor
alerts.sources.a1.interceptors.i1.regex = com.foo.(\\S*).*
alerts.sources.a1.interceptors.i1.serializers = s1
alerts.sources.a1.interceptors.i1.serializers.s1.name = app
{quote}
... with the following Log4j2 appender:
{quote}
<Flume name="eventLogger" suppressExceptions="false" compress="false">
<Agent host="localhost" port="4141"/>
<RFC5424Layout mdcId="bar" appName="com.foo.bar"/>
</Flume>
{quote}
... the interceptor will NOT insert the "app=bar" header. Instead I get a timestamp header. I understand the timestamp header, but I don't understand why the app header is missing, since the output of the logs looks something like this:
{quote}
<134>1 2013-05-08T16:11:28.836-04:00 host1 com.foo.bar - - [mdc@18060] Did something interesting
{quote}
In fact, if I configure a netcat source and send this string directly through the SAME interceptor, the header is added as expected.
As a further test, I configured an Avro sink and forward the event to another agent, where I then set up the same interceptor. In that case, both the app and timestamp headers were populated.
So to summarize:
FlumeAppender -> Avro source -> Regex interceptor = timestamp header
Raw string input -> NetCat source -> Regex interceptor = app header
FlumeAppender -> Avro source -> Avro sink -> Avro source -> Regex interceptor = both headers
was:
When using a source configured as follows:
alerts.sources.a1.type = avro
alerts.sources.a1.channels = c1
alerts.sources.a1.bind = 0.0.0.0
alerts.sources.a1.port = 4141
alerts.sources.a1.interceptors = i1
alerts.sources.a1.interceptors.i1.type = regex_extractor
alerts.sources.a1.interceptors.i1.regex = com.foo.(\\S*).*
alerts.sources.a1.interceptors.i1.serializers = s1
alerts.sources.a1.interceptors.i1.serializers.s1.name = app
... with the following Log4j2 appender:
<Flume name="eventLogger" suppressExceptions="false" compress="false">
<Agent host="localhost" port="4141"/>
<RFC5424Layout mdcId="bar" appName="com.foo.bar"/>
</Flume>
... the interceptor will NOT insert the "app=bar" header. Instead I get a timestamp header. I understand the timestamp header, but I don't understand why the app header is missing, since the output of the logs looks something like this:
<134>1 2013-05-08T16:11:28.836-04:00 host1 com.foo.bar - - [mdc@18060] Did something interesting
In fact, if I configure a netcat source and send this string directly through the SAME interceptor, the header is added as expected.
As a further test, I configured an Avro sink and forward the event to another agent, where I then set up the same interceptor. In that case, both the app and timestamp headers were populated.
So to summarize:
FlumeAppender -> Avro source -> Regex interceptor = timestamp header
Raw string input -> NetCat source -> Regex interceptor = app header
FlumeAppender -> Avro source -> Avro sink -> Avro source -> Regex interceptor = both headers
> Regex interceptor does not populate headers when using Log4j 2 FlumeAppender
> ----------------------------------------------------------------------------
>
> Key: FLUME-2041
> URL: https://issues.apache.org/jira/browse/FLUME-2041
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: NG alpha 2
> Reporter: Robbie Strickland
> Priority: Minor
> Labels: Log4j2
>
> When using a source configured as follows:
> {quote}
> alerts.sources.a1.type = avro
> alerts.sources.a1.channels = c1
> alerts.sources.a1.bind = 0.0.0.0
> alerts.sources.a1.port = 4141
> alerts.sources.a1.interceptors = i1
> alerts.sources.a1.interceptors.i1.type = regex_extractor
> alerts.sources.a1.interceptors.i1.regex = com.foo.(\\S*).*
> alerts.sources.a1.interceptors.i1.serializers = s1
> alerts.sources.a1.interceptors.i1.serializers.s1.name = app
> {quote}
> ... with the following Log4j2 appender:
> {quote}
> <Flume name="eventLogger" suppressExceptions="false" compress="false">
> <Agent host="localhost" port="4141"/>
> <RFC5424Layout mdcId="bar" appName="com.foo.bar"/>
> </Flume>
> {quote}
> ... the interceptor will NOT insert the "app=bar" header. Instead I get a timestamp header. I understand the timestamp header, but I don't understand why the app header is missing, since the output of the logs looks something like this:
> {quote}
> <134>1 2013-05-08T16:11:28.836-04:00 host1 com.foo.bar - - [mdc@18060] Did something interesting
> {quote}
> In fact, if I configure a netcat source and send this string directly through the SAME interceptor, the header is added as expected.
> As a further test, I configured an Avro sink and forward the event to another agent, where I then set up the same interceptor. In that case, both the app and timestamp headers were populated.
> So to summarize:
> FlumeAppender -> Avro source -> Regex interceptor = timestamp header
> Raw string input -> NetCat source -> Regex interceptor = app header
> FlumeAppender -> Avro source -> Avro sink -> Avro source -> Regex interceptor = both headers
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira