[jira] [Commented] (FLUME-1357) Strange host directory when using %{host}

["Juhani Connolly (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410029#comment-13410029 ] 

Juhani Connolly commented on FLUME-1357:
----------------------------------------

To be able to use %{host} you need to have the "host" header included in your events.

Similarly, to use the %Y %m %d replaces, you need the "timestamp" header.

You can add these using the timestamp and host interceptors to your source:

sender1.sources.tail.interceptors = ts host
sender1.sources.tail.interceptors.ts.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
sender1.sources.tail.interceptors.host.type = org.apache.flume.interceptor.HostInterceptor$Builder


                
> Strange host directory when using %{host}
> -----------------------------------------
>
>                 Key: FLUME-1357
>                 URL: https://issues.apache.org/jira/browse/FLUME-1357
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.1.0
>         Environment: - CentOS6.2
> - HDFS 2.0.0-cdh4.0.0
>            Reporter: Sebastien N.
>            Priority: Minor
>         Attachments: MARK-1.png, MARK-2.png
>
>
> Hi
> When i use: 
> agent.sources.src.type = syslogtcp
> agent.sources.src.port = 5140
> agent.sources.src.host = 0.0.0.0
> agent.sinks.s.type = hdfs
> agent.sinks.s.hdfs.path = hdfs://mycluster:8020/user/flume/%{host}/
> agent.sinks.s.serialize= org.apache.flume.serialization.BodyTextEventSerializer
> i have my list of rsyslog host and a special host named "MARK". 
> This directory contains files and the content of files is "--".
> I think this bug generate the following error when using 
> agent.sinks.s.hdfs.path = hdfs://mycluster:8020/user/flume/%{host}/%Y-%m-%d.
>  ERROR hdfs.HDFSEventSink: process failed
> java.lang.NumberFormatException: null
>         at java.lang.Long.parseLong(Long.java:404)
>         at java.lang.Long.valueOf(Long.java:540)
>         at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
>         at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> The headder of this event doesn't seem correct...
> More informations: 
> [root@mymachine:~]#hdfs dfs -copyToLocal /user/flume/MARK/SyslogTCP.1341818888517
> [root@mymachine:~]#cat SyslogTCP.1341818888517
> --

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira