create hdfs path

[kishore alajangi <al...@gmail.com>]

HI Experts,

My Avro file contains fields year, date and day. how to set the hdfs path
to create year, date and day partitions.Please help me.
-- 
Thanks,
Kishore.

Re: create hdfs path

[Matt Tenenbaum <ma...@rockyou.com>]

There may be even simpler ways, but you can implement
org.apache.flume.interceptor.Interceptor to construct a timestamp
(milliseconds) from those fields, and then set an event header for
"timestamp".

private static final String TIMESTAMP = "timestamp";

@Override
public Event intercept(Event event) {
    Map<String, String> headers = event.getHeaders();
    long ts = getTimestampMillis(event);
    headers.put(TIMESTAMP, Long.toString(ts));
    return event;
}

private long getTimestampMillis(Event event) {

    // construct timestamp-ms from avro fields
}


As a concrete example, I do this for protobuf-encoded data (I imagine
handling avro data is similar, though I haven't used avro to represent
events in flume). I need to deserialize the protobuf representation so that
I can extract a field holding the time as a standard unix timestamp. My
implementation of getTimestampMillis would look like this:

private long getTimestampMillis(Event event) {
    LogMessage message = LogMessage.parseFrom(event.getBody());
    return message.getTimestamp() * 1000;
}

Once a timestamp header exists, you can simply specify the formatting you
want for the path and it will write each event into a file under the
correct path for that day. In your configuration:

a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d


will create a directory for each day, to hold files for all the events
whose avro fields reflect that particular day.

-mt





On Sun, Jun 29, 2014 at 12:31 AM, kishore alajangi <
alajangikishore@gmail.com> wrote:

>
> HI Experts,
>
> My Avro file contains fields year, date and day. how to set the hdfs path
> to create year, date and day partitions.Please help me.
> --
> Thanks,
> Kishore.
>