HDFS custom escape sequence

["Camp, Roy" <rc...@ebay.com>]

Is it possible to use other headers besides the host header, %{host} escape sequence, to name the HDFS files?  We are interested in attaching a category header and want the events stored separately.

If not possible, is there a best practice for achieving this?

Thanks,

Roy

Re: HDFS custom escape sequence

[Hari Shreedharan <hs...@cloudera.com>]

You should be able to use %{headerName} to replace the escape sequence in the HDFS sink. For example if you have a header "topic" with value "flume", an HDFS path that looks like: 

hdfs://nameservice/user/flume/%{topic} will drop the event into user/flume/flume. 


Cheers,
Hari


On Monday, June 10, 2013 at 2:51 PM, Camp, Roy wrote:

> Is it possible to use other headers besides the host header, %{host} escape sequence, to name the HDFS files?  We are interested in attaching a category header and want the events stored separately.  
>  
> If not possible, is there a best practice for achieving this?
>  
> Thanks,
>  
> Roy
>  
>  
> 
> 
> 

RE: HDFS custom escape sequence

[Paul Chavez <pc...@verticalsearchworks.com>]

Yes, any header works using that same notation. I use it to allow us to bucket many event 'types' with a single channel.

for example, a path with custom headers and Hive ready partition folders:
/flumelogs/%{someHeader}/%{someOtherHeader}/datekey=%Y%m%d/hour=%H

Hope that helps,
Paul Chavez

________________________________
From: Camp, Roy [mailto:rcamp@ebay.com]
Sent: Monday, June 10, 2013 2:51 PM
To: user@flume.apache.org
Subject: HDFS custom escape sequence

Is it possible to use other headers besides the host header, %{host} escape sequence, to name the HDFS files?  We are interested in attaching a category header and want the events stored separately.

If not possible, is there a best practice for achieving this?

Thanks,

Roy