[jira] [Created] (FLUME-2041) Regex interceptor does not populate headers when using Log4j 2 FlumeAppender

["Robbie Strickland (JIRA)" <ji...@apache.org>]

Robbie Strickland created FLUME-2041:
----------------------------------------

             Summary: Regex interceptor does not populate headers when using Log4j 2 FlumeAppender
                 Key: FLUME-2041
                 URL: https://issues.apache.org/jira/browse/FLUME-2041
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: NG alpha 2
            Reporter: Robbie Strickland
            Priority: Minor


When using a source configured as follows:

alerts.sources.a1.type = avro
alerts.sources.a1.channels = c1
alerts.sources.a1.bind = 0.0.0.0
alerts.sources.a1.port = 4141

alerts.sources.a1.interceptors = i1
alerts.sources.a1.interceptors.i1.type = regex_extractor
alerts.sources.a1.interceptors.i1.regex = com.foo.(\\S*).*
alerts.sources.a1.interceptors.i1.serializers = s1
alerts.sources.a1.interceptors.i1.serializers.s1.name = app

... with the following Log4j2 appender:

<Flume name="eventLogger" suppressExceptions="false" compress="false">
  <Agent host="localhost" port="4141"/>
  <RFC5424Layout mdcId="bar" appName="com.foo.bar"/>
</Flume>

... the interceptor will NOT insert the "app=bar" header.  Instead I get a timestamp header.  I understand the timestamp header, but I don't understand why the app header is missing, since the output of the logs looks something like this:

<134>1 2013-05-08T16:11:28.836-04:00 host1 com.foo.bar - - [mdc@18060] Did something interesting

In fact, if I configure a netcat source and send this string directly through the SAME interceptor, the header is added as expected.

As a further test, I configured an Avro sink and forward the event to another agent, where I then set up the same interceptor.  In that case, both the app and timestamp headers were populated.

So to summarize:
FlumeAppender -> Avro source -> Regex interceptor = timestamp header
Raw string input -> NetCat source -> Regex interceptor = app header
FlumeAppender -> Avro source -> Avro sink -> Avro source -> Regex interceptor = both headers


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira