You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Mohit Anchlia <mo...@gmail.com> on 2012/08/22 02:52:32 UTC
HDFS escape sequence and use of timestamp header
I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do I
need to write some custom interceptor? Could you please point me to an
example? Currently my AvroClient looks like this:
*public* *void* sendDataToFlume(String data) {
// Create *flume* event object
Event event = EventBuilder.*withBody*(data, Charset.*forName*("UTF-8"));
Map<String,String> headers = *new* HashMap<String,String>();
headers.put("host", *hostName*);
event.setHeaders(headers);
*try* {
rpcClient.append(event);
} *catch* (EventDeliveryException e) {
connect();
}
}
Re: HDFS escape sequence and use of timestamp header
Posted by Mohit Anchlia <mo...@gmail.com>.
thanks that works
On Wed, Aug 22, 2012 at 5:52 PM, Kathleen Ting <ka...@apache.org> wrote:
> Mohit,
>
> As it happens, both the example and the table are correct. You can
> specify either the fully qualified class name or the alias 'TIMESTAMP'
>
> Regards, Kathleen
>
> On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
> >
> >
> > On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <ka...@apache.org>
> wrote:
> >>
> >> Hi Mohit,
> >>
> >> You can configure a timestamp interceptor onto your source as follows:
> >>
> >> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
> >>
> >> Once the timestamp interceptor is in place, you can use, in sinks, the
> >> timestamp it writes. Here is an example of using the timestamp in an
> >> HDFS sink:
> >>
> >> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
> >>
> >> Here is more info about the timestamp interceptor:
> >> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
> >
> > Thanks this helps. The type specified in the example is a fully qualified
> > class name, however in the table it says use type as TIMESTAMP. I guess
> > example is not correct and I should just use TIMESTAMP.
> >>
> >> Regards, Kathleen
> >>
> >> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <mohitanchlia@gmail.com
> >
> >> wrote:
> >> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but
> how
> >> > do
> >> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I
> just
> >> > get
> >> > number format exceptions.
> >> >
> >> >
> >> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <
> mohitanchlia@gmail.com>
> >> > wrote:
> >> >>
> >> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence.
> Do
> >> >> I
> >> >> need to write some custom interceptor? Could you please point me to
> an
> >> >> example? Currently my AvroClient looks like this:
> >> >>
> >> >>
> >> >> public void sendDataToFlume(String data) {
> >> >>
> >> >> // Create flume event object
> >> >>
> >> >> Event event = EventBuilder.withBody(data, Charset.forName(
> >> >>
> >> >> "UTF-8"));
> >> >>
> >> >> Map<String,String> headers =
> >> >>
> >> >> new HashMap<String,String>();
> >> >>
> >> >> headers.put(
> >> >>
> >> >> "host", hostName);
> >> >>
> >> >> event.setHeaders(headers);
> >> >>
> >> >> try {
> >> >>
> >> >> rpcClient.append(event);
> >> >>
> >> >> }
> >> >>
> >> >> catch (EventDeliveryException e) {
> >> >>
> >> >> connect();
> >> >>
> >> >> }
> >> >>
> >> >> }
> >> >
> >> >
> >
> >
>
Re: HDFS escape sequence and use of timestamp header
Posted by Kathleen Ting <ka...@apache.org>.
Mohit,
As it happens, both the example and the table are correct. You can
specify either the fully qualified class name or the alias 'TIMESTAMP'
Regards, Kathleen
On Wed, Aug 22, 2012 at 5:05 PM, Mohit Anchlia <mo...@gmail.com> wrote:
>
>
> On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <ka...@apache.org> wrote:
>>
>> Hi Mohit,
>>
>> You can configure a timestamp interceptor onto your source as follows:
>>
>> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
>>
>> Once the timestamp interceptor is in place, you can use, in sinks, the
>> timestamp it writes. Here is an example of using the timestamp in an
>> HDFS sink:
>>
>> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
>>
>> Here is more info about the timestamp interceptor:
>> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
>
> Thanks this helps. The type specified in the example is a fully qualified
> class name, however in the table it says use type as TIMESTAMP. I guess
> example is not correct and I should just use TIMESTAMP.
>>
>> Regards, Kathleen
>>
>> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <mo...@gmail.com>
>> wrote:
>> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but how
>> > do
>> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I just
>> > get
>> > number format exceptions.
>> >
>> >
>> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <mo...@gmail.com>
>> > wrote:
>> >>
>> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do
>> >> I
>> >> need to write some custom interceptor? Could you please point me to an
>> >> example? Currently my AvroClient looks like this:
>> >>
>> >>
>> >> public void sendDataToFlume(String data) {
>> >>
>> >> // Create flume event object
>> >>
>> >> Event event = EventBuilder.withBody(data, Charset.forName(
>> >>
>> >> "UTF-8"));
>> >>
>> >> Map<String,String> headers =
>> >>
>> >> new HashMap<String,String>();
>> >>
>> >> headers.put(
>> >>
>> >> "host", hostName);
>> >>
>> >> event.setHeaders(headers);
>> >>
>> >> try {
>> >>
>> >> rpcClient.append(event);
>> >>
>> >> }
>> >>
>> >> catch (EventDeliveryException e) {
>> >>
>> >> connect();
>> >>
>> >> }
>> >>
>> >> }
>> >
>> >
>
>
Re: HDFS escape sequence and use of timestamp header
Posted by Mohit Anchlia <mo...@gmail.com>.
On Wed, Aug 22, 2012 at 2:16 PM, Kathleen Ting <ka...@apache.org> wrote:
> Hi Mohit,
>
> You can configure a timestamp interceptor onto your source as follows:
>
> agent.sources.src-0.interceptors.ts.type = TIMESTAMP
>
> Once the timestamp interceptor is in place, you can use, in sinks, the
> timestamp it writes. Here is an example of using the timestamp in an
> HDFS sink:
>
> agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
>
> Here is more info about the timestamp interceptor:
> http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
>
Thanks this helps. The type specified in the example is a fully qualified
class name, however in the table it says use type as TIMESTAMP. I guess
example is not correct and I should just use TIMESTAMP.
> Regards, Kathleen
>
> On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <mo...@gmail.com>
> wrote:
> > I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but
> how do
> > I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I just
> get
> > number format exceptions.
> >
> >
> > On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <mo...@gmail.com>
> > wrote:
> >>
> >> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do I
> >> need to write some custom interceptor? Could you please point me to an
> >> example? Currently my AvroClient looks like this:
> >>
> >>
> >> public void sendDataToFlume(String data) {
> >>
> >> // Create flume event object
> >>
> >> Event event = EventBuilder.withBody(data, Charset.forName(
> >>
> >> "UTF-8"));
> >>
> >> Map<String,String> headers =
> >>
> >> new HashMap<String,String>();
> >>
> >> headers.put(
> >>
> >> "host", hostName);
> >>
> >> event.setHeaders(headers);
> >>
> >> try {
> >>
> >> rpcClient.append(event);
> >>
> >> }
> >>
> >> catch (EventDeliveryException e) {
> >>
> >> connect();
> >>
> >> }
> >>
> >> }
> >
> >
>
Re: HDFS escape sequence and use of timestamp header
Posted by Kathleen Ting <ka...@apache.org>.
Hi Mohit,
You can configure a timestamp interceptor onto your source as follows:
agent.sources.src-0.interceptors.ts.type = TIMESTAMP
Once the timestamp interceptor is in place, you can use, in sinks, the
timestamp it writes. Here is an example of using the timestamp in an
HDFS sink:
agent.sinks.sink-0.hdfs.filePrefix = FlumeData.%Y-%m-%d
Here is more info about the timestamp interceptor:
http://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor
Regards, Kathleen
On Wed, Aug 22, 2012 at 11:50 AM, Mohit Anchlia <mo...@gmail.com> wrote:
> I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but how do
> I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I just get
> number format exceptions.
>
>
> On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
>>
>> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do I
>> need to write some custom interceptor? Could you please point me to an
>> example? Currently my AvroClient looks like this:
>>
>>
>> public void sendDataToFlume(String data) {
>>
>> // Create flume event object
>>
>> Event event = EventBuilder.withBody(data, Charset.forName(
>>
>> "UTF-8"));
>>
>> Map<String,String> headers =
>>
>> new HashMap<String,String>();
>>
>> headers.put(
>>
>> "host", hostName);
>>
>> event.setHeaders(headers);
>>
>> try {
>>
>> rpcClient.append(event);
>>
>> }
>>
>> catch (EventDeliveryException e) {
>>
>> connect();
>>
>> }
>>
>> }
>
>
Re: HDFS escape sequence and use of timestamp header
Posted by Mohit Anchlia <mo...@gmail.com>.
I see this JIRA https://issues.apache.org/jira/browse/FLUME-1215 but how do
I take advantage of it? I am using 1.2.0 but %Y %m doesn't work. I just get
number format exceptions.
On Tue, Aug 21, 2012 at 5:52 PM, Mohit Anchlia <mo...@gmail.com>wrote:
> I am using flume-ng 1.2.0 and I need to use %Y%M%D escape sequence. Do I
> need to write some custom interceptor? Could you please point me to an
> example? Currently my AvroClient looks like this:
>
>
> *public* *void* sendDataToFlume(String data) {
>
> // Create *flume* event object
>
> Event event = EventBuilder.*withBody*(data, Charset.*forName*(
> "UTF-8"));
>
> Map<String,String> headers =
> *new* HashMap<String,String>();
>
> headers.put(
> "host", *hostName*);
>
> event.setHeaders(headers);
>
> *try* {
>
> rpcClient.append(event);
>
> }
> *catch* (EventDeliveryException e) {
>
> connect();
>
> }
>
> }
>