You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Martinus m <ma...@gmail.com> on 2013/10/08 12:45:55 UTC

send the whole logs

Hi,

What configurations should I put on flume.conf to get the whole logs
message to be put into the sink?

Thanks.

Martinus

Re: send the whole logs

Posted by Martinus m <ma...@gmail.com>.
Hi David & Jeff,

Thanks to both of you. I used the Flume-1.5.0-SNAPSHOT version and it's
worked.

Thanks.

Martinus


On Sat, Oct 12, 2013 at 2:05 PM, Jeff Lord <jl...@cloudera.com> wrote:

> So if you use trunk and set the keepFields property to true than the
> Timestamp and Hostname will be preserved in the body of the event now.
>
>
> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst#syslog-sources
>
>
> On Fri, Oct 11, 2013 at 7:29 AM, David Sinclair <
> dsinclair@chariotsolutions.com> wrote:
>
>> Ah, now I understand. The syslog source is currently parsing out some of
>> the fields and putting them in as headers, e.g. facility, severity,
>> timestamp, hostname.
>>
>> If you want to have it output in the original format, you can implement
>> a EventSerializer.  You could take a look at SyslogAvroEventSerializer to
>> see how it deals with the syslog Flume event. Or implement your
>> own SyslogUDPSource that puts the entire message in the Flume event's body.
>>
>> dave
>>
>>
>> On Fri, Oct 11, 2013 at 3:36 AM, Martinus m <ma...@gmail.com>wrote:
>>
>>> Hi David,
>>>
>>> Actually the requirement is I need to send the whole logs info as the
>>> original one, before someone else do the MR on it. Is there any other
>>> options in Flume configurations that I can do this?
>>>
>>> Thanks.
>>>
>>> Martinus
>>>
>>>
>>> On Wed, Oct 9, 2013 at 8:53 PM, David Sinclair <
>>> dsinclair@chariotsolutions.com> wrote:
>>>
>>>> That is the original timestamp; just in seconds since epoch, not
>>>> formatted as a string. Could you parse that in MR to a date if you need to
>>>> manipulate it as such?
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 3:52 AM, Martinus m <ma...@gmail.com>wrote:
>>>>
>>>>> Hi Hari,
>>>>>
>>>>> Thanks, it's worked, but it's timestamp information doesn't looks like
>>>>> the original one :
>>>>>
>>>>> {timestamp=1381304766000, host=flume, Severity=6, Facility=3}
>>>>>
>>>>> The original one is like below :
>>>>>
>>>>> Oct  9 07:46:06 flume
>>>>>
>>>>> Is there any other configuration that I miss to make this header just
>>>>> looks like the original message?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Martinus
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 3:29 PM, Hari Shreedharan <
>>>>> hshreedharan@cloudera.com> wrote:
>>>>>
>>>>>>  text does not write the headers, try HEADER_AND_TEXT
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Hari
>>>>>>
>>>>>> On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:
>>>>>>
>>>>>> Hi Hari,
>>>>>>
>>>>>> I tried to add below serializers in my flume.conf :
>>>>>>
>>>>>> agent.sinks.s3Sink.serializer = text
>>>>>>
>>>>>> And it's still doesn't have timestamp (date) info from the original
>>>>>> log message :
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Martinus
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <
>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>
>>>>>>  The timestamp is in the event header. You would need to use a
>>>>>> serializer which also writes out the headers.
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Hari
>>>>>>
>>>>>> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>>>>>>
>>>>>> Hi David,
>>>>>>
>>>>>> I'm using Syslog UDP source and on the syslog messages it have it's
>>>>>> timestamp for each message. I also use HDFS sink, but when I saw the result
>>>>>> message on HDFS folder, it doesn't have the timestamp (date) info.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Martinus
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
>>>>>> dsinclair@chariotsolutions.com> wrote:
>>>>>>
>>>>>> Martinus,
>>>>>>
>>>>>> Can you give a little more details? It sounds like you want to use
>>>>>> the Spooling Directory Source,
>>>>>> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
>>>>>> but if you can be clearer about your requirements, I may be able to help
>>>>>> you better.
>>>>>>
>>>>>> dave
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com>wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What configurations should I put on flume.conf to get the whole logs
>>>>>> message to be put into the sink?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Martinus
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: send the whole logs

Posted by Jeff Lord <jl...@cloudera.com>.
So if you use trunk and set the keepFields property to true than the
Timestamp and Hostname will be preserved in the body of the event now.

https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst#syslog-sources


On Fri, Oct 11, 2013 at 7:29 AM, David Sinclair <
dsinclair@chariotsolutions.com> wrote:

> Ah, now I understand. The syslog source is currently parsing out some of
> the fields and putting them in as headers, e.g. facility, severity,
> timestamp, hostname.
>
> If you want to have it output in the original format, you can implement
> a EventSerializer.  You could take a look at SyslogAvroEventSerializer to
> see how it deals with the syslog Flume event. Or implement your
> own SyslogUDPSource that puts the entire message in the Flume event's body.
>
> dave
>
>
> On Fri, Oct 11, 2013 at 3:36 AM, Martinus m <ma...@gmail.com> wrote:
>
>> Hi David,
>>
>> Actually the requirement is I need to send the whole logs info as the
>> original one, before someone else do the MR on it. Is there any other
>> options in Flume configurations that I can do this?
>>
>> Thanks.
>>
>> Martinus
>>
>>
>> On Wed, Oct 9, 2013 at 8:53 PM, David Sinclair <
>> dsinclair@chariotsolutions.com> wrote:
>>
>>> That is the original timestamp; just in seconds since epoch, not
>>> formatted as a string. Could you parse that in MR to a date if you need to
>>> manipulate it as such?
>>>
>>>
>>> On Wed, Oct 9, 2013 at 3:52 AM, Martinus m <ma...@gmail.com>wrote:
>>>
>>>> Hi Hari,
>>>>
>>>> Thanks, it's worked, but it's timestamp information doesn't looks like
>>>> the original one :
>>>>
>>>> {timestamp=1381304766000, host=flume, Severity=6, Facility=3}
>>>>
>>>> The original one is like below :
>>>>
>>>> Oct  9 07:46:06 flume
>>>>
>>>> Is there any other configuration that I miss to make this header just
>>>> looks like the original message?
>>>>
>>>> Thanks,
>>>>
>>>> Martinus
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 3:29 PM, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>>>  text does not write the headers, try HEADER_AND_TEXT
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:
>>>>>
>>>>> Hi Hari,
>>>>>
>>>>> I tried to add below serializers in my flume.conf :
>>>>>
>>>>> agent.sinks.s3Sink.serializer = text
>>>>>
>>>>> And it's still doesn't have timestamp (date) info from the original
>>>>> log message :
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Martinus
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <
>>>>> hshreedharan@cloudera.com> wrote:
>>>>>
>>>>>  The timestamp is in the event header. You would need to use a
>>>>> serializer which also writes out the headers.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>>>>>
>>>>> Hi David,
>>>>>
>>>>> I'm using Syslog UDP source and on the syslog messages it have it's
>>>>> timestamp for each message. I also use HDFS sink, but when I saw the result
>>>>> message on HDFS folder, it doesn't have the timestamp (date) info.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Martinus
>>>>>
>>>>>
>>>>> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
>>>>> dsinclair@chariotsolutions.com> wrote:
>>>>>
>>>>> Martinus,
>>>>>
>>>>> Can you give a little more details? It sounds like you want to use the
>>>>> Spooling Directory Source,
>>>>> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
>>>>> but if you can be clearer about your requirements, I may be able to help
>>>>> you better.
>>>>>
>>>>> dave
>>>>>
>>>>>
>>>>> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com>wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> What configurations should I put on flume.conf to get the whole logs
>>>>> message to be put into the sink?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Martinus
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: send the whole logs

Posted by David Sinclair <ds...@chariotsolutions.com>.
Ah, now I understand. The syslog source is currently parsing out some of
the fields and putting them in as headers, e.g. facility, severity,
timestamp, hostname.

If you want to have it output in the original format, you can implement
a EventSerializer.  You could take a look at SyslogAvroEventSerializer to
see how it deals with the syslog Flume event. Or implement your
own SyslogUDPSource that puts the entire message in the Flume event's body.

dave


On Fri, Oct 11, 2013 at 3:36 AM, Martinus m <ma...@gmail.com> wrote:

> Hi David,
>
> Actually the requirement is I need to send the whole logs info as the
> original one, before someone else do the MR on it. Is there any other
> options in Flume configurations that I can do this?
>
> Thanks.
>
> Martinus
>
>
> On Wed, Oct 9, 2013 at 8:53 PM, David Sinclair <
> dsinclair@chariotsolutions.com> wrote:
>
>> That is the original timestamp; just in seconds since epoch, not
>> formatted as a string. Could you parse that in MR to a date if you need to
>> manipulate it as such?
>>
>>
>> On Wed, Oct 9, 2013 at 3:52 AM, Martinus m <ma...@gmail.com> wrote:
>>
>>> Hi Hari,
>>>
>>> Thanks, it's worked, but it's timestamp information doesn't looks like
>>> the original one :
>>>
>>> {timestamp=1381304766000, host=flume, Severity=6, Facility=3}
>>>
>>> The original one is like below :
>>>
>>> Oct  9 07:46:06 flume
>>>
>>> Is there any other configuration that I miss to make this header just
>>> looks like the original message?
>>>
>>> Thanks,
>>>
>>> Martinus
>>>
>>>
>>> On Wed, Oct 9, 2013 at 3:29 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>>  text does not write the headers, try HEADER_AND_TEXT
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:
>>>>
>>>> Hi Hari,
>>>>
>>>> I tried to add below serializers in my flume.conf :
>>>>
>>>> agent.sinks.s3Sink.serializer = text
>>>>
>>>> And it's still doesn't have timestamp (date) info from the original log
>>>> message :
>>>>
>>>> Thanks,
>>>>
>>>> Martinus
>>>>
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <
>>>> hshreedharan@cloudera.com> wrote:
>>>>
>>>>  The timestamp is in the event header. You would need to use a
>>>> serializer which also writes out the headers.
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>>>>
>>>> Hi David,
>>>>
>>>> I'm using Syslog UDP source and on the syslog messages it have it's
>>>> timestamp for each message. I also use HDFS sink, but when I saw the result
>>>> message on HDFS folder, it doesn't have the timestamp (date) info.
>>>>
>>>> Thanks,
>>>>
>>>> Martinus
>>>>
>>>>
>>>> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
>>>> dsinclair@chariotsolutions.com> wrote:
>>>>
>>>> Martinus,
>>>>
>>>> Can you give a little more details? It sounds like you want to use the
>>>> Spooling Directory Source,
>>>> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
>>>> but if you can be clearer about your requirements, I may be able to help
>>>> you better.
>>>>
>>>> dave
>>>>
>>>>
>>>> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com>wrote:
>>>>
>>>> Hi,
>>>>
>>>> What configurations should I put on flume.conf to get the whole logs
>>>> message to be put into the sink?
>>>>
>>>> Thanks.
>>>>
>>>> Martinus
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

Re: send the whole logs

Posted by Martinus m <ma...@gmail.com>.
Hi David,

Actually the requirement is I need to send the whole logs info as the
original one, before someone else do the MR on it. Is there any other
options in Flume configurations that I can do this?

Thanks.

Martinus


On Wed, Oct 9, 2013 at 8:53 PM, David Sinclair <
dsinclair@chariotsolutions.com> wrote:

> That is the original timestamp; just in seconds since epoch, not formatted
> as a string. Could you parse that in MR to a date if you need to manipulate
> it as such?
>
>
> On Wed, Oct 9, 2013 at 3:52 AM, Martinus m <ma...@gmail.com> wrote:
>
>> Hi Hari,
>>
>> Thanks, it's worked, but it's timestamp information doesn't looks like
>> the original one :
>>
>> {timestamp=1381304766000, host=flume, Severity=6, Facility=3}
>>
>> The original one is like below :
>>
>> Oct  9 07:46:06 flume
>>
>> Is there any other configuration that I miss to make this header just
>> looks like the original message?
>>
>> Thanks,
>>
>> Martinus
>>
>>
>> On Wed, Oct 9, 2013 at 3:29 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>>  text does not write the headers, try HEADER_AND_TEXT
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:
>>>
>>> Hi Hari,
>>>
>>> I tried to add below serializers in my flume.conf :
>>>
>>> agent.sinks.s3Sink.serializer = text
>>>
>>> And it's still doesn't have timestamp (date) info from the original log
>>> message :
>>>
>>> Thanks,
>>>
>>> Martinus
>>>
>>>
>>>
>>> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <
>>> hshreedharan@cloudera.com> wrote:
>>>
>>>  The timestamp is in the event header. You would need to use a
>>> serializer which also writes out the headers.
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>>>
>>> Hi David,
>>>
>>> I'm using Syslog UDP source and on the syslog messages it have it's
>>> timestamp for each message. I also use HDFS sink, but when I saw the result
>>> message on HDFS folder, it doesn't have the timestamp (date) info.
>>>
>>> Thanks,
>>>
>>> Martinus
>>>
>>>
>>> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
>>> dsinclair@chariotsolutions.com> wrote:
>>>
>>> Martinus,
>>>
>>> Can you give a little more details? It sounds like you want to use the
>>> Spooling Directory Source,
>>> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
>>> but if you can be clearer about your requirements, I may be able to help
>>> you better.
>>>
>>> dave
>>>
>>>
>>> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com>wrote:
>>>
>>> Hi,
>>>
>>> What configurations should I put on flume.conf to get the whole logs
>>> message to be put into the sink?
>>>
>>> Thanks.
>>>
>>> Martinus
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Re: send the whole logs

Posted by David Sinclair <ds...@chariotsolutions.com>.
That is the original timestamp; just in seconds since epoch, not formatted
as a string. Could you parse that in MR to a date if you need to manipulate
it as such?


On Wed, Oct 9, 2013 at 3:52 AM, Martinus m <ma...@gmail.com> wrote:

> Hi Hari,
>
> Thanks, it's worked, but it's timestamp information doesn't looks like the
> original one :
>
> {timestamp=1381304766000, host=flume, Severity=6, Facility=3}
>
> The original one is like below :
>
> Oct  9 07:46:06 flume
>
> Is there any other configuration that I miss to make this header just
> looks like the original message?
>
> Thanks,
>
> Martinus
>
>
> On Wed, Oct 9, 2013 at 3:29 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>>  text does not write the headers, try HEADER_AND_TEXT
>>
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:
>>
>> Hi Hari,
>>
>> I tried to add below serializers in my flume.conf :
>>
>> agent.sinks.s3Sink.serializer = text
>>
>> And it's still doesn't have timestamp (date) info from the original log
>> message :
>>
>> Thanks,
>>
>> Martinus
>>
>>
>>
>> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>  The timestamp is in the event header. You would need to use a serializer
>> which also writes out the headers.
>>
>>
>> Thanks,
>> Hari
>>
>> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>>
>> Hi David,
>>
>> I'm using Syslog UDP source and on the syslog messages it have it's
>> timestamp for each message. I also use HDFS sink, but when I saw the result
>> message on HDFS folder, it doesn't have the timestamp (date) info.
>>
>> Thanks,
>>
>> Martinus
>>
>>
>> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
>> dsinclair@chariotsolutions.com> wrote:
>>
>> Martinus,
>>
>> Can you give a little more details? It sounds like you want to use the
>> Spooling Directory Source,
>> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
>> but if you can be clearer about your requirements, I may be able to help
>> you better.
>>
>> dave
>>
>>
>> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com> wrote:
>>
>> Hi,
>>
>> What configurations should I put on flume.conf to get the whole logs
>> message to be put into the sink?
>>
>> Thanks.
>>
>> Martinus
>>
>>
>>
>>
>>
>>
>>
>

Re: send the whole logs

Posted by Martinus m <ma...@gmail.com>.
Hi Hari,

Thanks, it's worked, but it's timestamp information doesn't looks like the
original one :

{timestamp=1381304766000, host=flume, Severity=6, Facility=3}

The original one is like below :

Oct  9 07:46:06 flume

Is there any other configuration that I miss to make this header just looks
like the original message?

Thanks,

Martinus


On Wed, Oct 9, 2013 at 3:29 PM, Hari Shreedharan
<hs...@cloudera.com>wrote:

>  text does not write the headers, try HEADER_AND_TEXT
>
>
> Thanks,
> Hari
>
> On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:
>
> Hi Hari,
>
> I tried to add below serializers in my flume.conf :
>
> agent.sinks.s3Sink.serializer = text
>
> And it's still doesn't have timestamp (date) info from the original log
> message :
>
> Thanks,
>
> Martinus
>
>
>
> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>  The timestamp is in the event header. You would need to use a serializer
> which also writes out the headers.
>
>
> Thanks,
> Hari
>
> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>
> Hi David,
>
> I'm using Syslog UDP source and on the syslog messages it have it's
> timestamp for each message. I also use HDFS sink, but when I saw the result
> message on HDFS folder, it doesn't have the timestamp (date) info.
>
> Thanks,
>
> Martinus
>
>
> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
> dsinclair@chariotsolutions.com> wrote:
>
> Martinus,
>
> Can you give a little more details? It sounds like you want to use the
> Spooling Directory Source,
> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
> but if you can be clearer about your requirements, I may be able to help
> you better.
>
> dave
>
>
> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com> wrote:
>
> Hi,
>
> What configurations should I put on flume.conf to get the whole logs
> message to be put into the sink?
>
> Thanks.
>
> Martinus
>
>
>
>
>
>
>

Re: send the whole logs

Posted by Hari Shreedharan <hs...@cloudera.com>.
text does not write the headers, try HEADER_AND_TEXT  


Thanks,
Hari


On Tuesday, October 8, 2013 at 11:26 PM, Martinus m wrote:

> Hi Hari,
> 
> I tried to add below serializers in my flume.conf :
> 
> agent.sinks.s3Sink.serializer = text
> 
> And it's still doesn't have timestamp (date) info from the original log message : 
> 
> Thanks,
> 
> Martinus
> 
> 
> 
> On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > The timestamp is in the event header. You would need to use a serializer which also writes out the headers. 
> > 
> > 
> > Thanks,
> > Hari
> > 
> > 
> > On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
> > 
> > > Hi David,
> > > 
> > > I'm using Syslog UDP source and on the syslog messages it have it's timestamp for each message. I also use HDFS sink, but when I saw the result message on HDFS folder, it doesn't have the timestamp (date) info. 
> > > 
> > > Thanks,
> > > 
> > > Martinus
> > > 
> > > 
> > > On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <dsinclair@chariotsolutions.com (mailto:dsinclair@chariotsolutions.com)> wrote:
> > > > Martinus,
> > > > 
> > > > Can you give a little more details? It sounds like you want to use the Spooling Directory Source, http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source, but if you can be clearer about your requirements, I may be able to help you better. 
> > > > 
> > > > dave
> > > > 
> > > > 
> > > > On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <martinus787@gmail.com (mailto:martinus787@gmail.com)> wrote:
> > > > > Hi,
> > > > > 
> > > > > What configurations should I put on flume.conf to get the whole logs message to be put into the sink? 
> > > > > 
> > > > > Thanks.
> > > > > 
> > > > > Martinus 
> > > 
> > 
> 


Re: send the whole logs

Posted by Martinus m <ma...@gmail.com>.
Hi Hari,

I tried to add below serializers in my flume.conf :

agent.sinks.s3Sink.serializer = text

And it's still doesn't have timestamp (date) info from the original log
message :

Thanks,

Martinus



On Wed, Oct 9, 2013 at 1:45 PM, Hari Shreedharan
<hs...@cloudera.com>wrote:

>  The timestamp is in the event header. You would need to use a serializer
> which also writes out the headers.
>
>
> Thanks,
> Hari
>
> On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:
>
> Hi David,
>
> I'm using Syslog UDP source and on the syslog messages it have it's
> timestamp for each message. I also use HDFS sink, but when I saw the result
> message on HDFS folder, it doesn't have the timestamp (date) info.
>
> Thanks,
>
> Martinus
>
>
> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
> dsinclair@chariotsolutions.com> wrote:
>
> Martinus,
>
> Can you give a little more details? It sounds like you want to use the
> Spooling Directory Source,
> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
> but if you can be clearer about your requirements, I may be able to help
> you better.
>
> dave
>
>
> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com> wrote:
>
> Hi,
>
> What configurations should I put on flume.conf to get the whole logs
> message to be put into the sink?
>
> Thanks.
>
> Martinus
>
>
>
>
>

Re: send the whole logs

Posted by Hari Shreedharan <hs...@cloudera.com>.
The timestamp is in the event header. You would need to use a serializer which also writes out the headers. 


Thanks,
Hari


On Tuesday, October 8, 2013 at 7:35 PM, Martinus m wrote:

> Hi David,
> 
> I'm using Syslog UDP source and on the syslog messages it have it's timestamp for each message. I also use HDFS sink, but when I saw the result message on HDFS folder, it doesn't have the timestamp (date) info. 
> 
> Thanks,
> 
> Martinus
> 
> 
> On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <dsinclair@chariotsolutions.com (mailto:dsinclair@chariotsolutions.com)> wrote:
> > Martinus,
> > 
> > Can you give a little more details? It sounds like you want to use the Spooling Directory Source, http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source, but if you can be clearer about your requirements, I may be able to help you better. 
> > 
> > dave
> > 
> > 
> > On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <martinus787@gmail.com (mailto:martinus787@gmail.com)> wrote:
> > > Hi,
> > > 
> > > What configurations should I put on flume.conf to get the whole logs message to be put into the sink? 
> > > 
> > > Thanks.
> > > 
> > > Martinus 
> 


Re: send the whole logs

Posted by Martinus m <ma...@gmail.com>.
Hi David,

I'm using Syslog UDP source and on the syslog messages it have it's
timestamp for each message. I also use HDFS sink, but when I saw the result
message on HDFS folder, it doesn't have the timestamp (date) info.

Thanks,

Martinus


On Tue, Oct 8, 2013 at 9:01 PM, David Sinclair <
dsinclair@chariotsolutions.com> wrote:

> Martinus,
>
> Can you give a little more details? It sounds like you want to use the
> Spooling Directory Source,
> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source,
> but if you can be clearer about your requirements, I may be able to help
> you better.
>
> dave
>
>
> On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com> wrote:
>
>> Hi,
>>
>> What configurations should I put on flume.conf to get the whole logs
>> message to be put into the sink?
>>
>> Thanks.
>>
>> Martinus
>>
>
>

Re: send the whole logs

Posted by David Sinclair <ds...@chariotsolutions.com>.
Martinus,

Can you give a little more details? It sounds like you want to use the
Spooling Directory Source,
http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source, but
if you can be clearer about your requirements, I may be able to help you
better.

dave


On Tue, Oct 8, 2013 at 6:45 AM, Martinus m <ma...@gmail.com> wrote:

> Hi,
>
> What configurations should I put on flume.conf to get the whole logs
> message to be put into the sink?
>
> Thanks.
>
> Martinus
>