You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by Jiaqi Tan <ta...@gmail.com> on 2009/05/18 03:15:09 UTC
Adaptor options for Hadoop log files (TaskTracker, DataNode)
Hi,
Which adaptor should I use if I want to process log entries from the
TaskTracker and DataNode logs? Should I just use one of the
FileTailer adaptors already available (CharFileTailingAdaptorUTF8), or
is there a custom type such as the one for SysLog or NameNodeLog when
using the CharFileTailingAdaptorUTF8NewLineEscaped adaptor?
Is there any documentation available on what the "type" (e.g. SysLog
or NameNodeLog) means and how to use it/how it works?
Thanks,
Jiaqi
Re: Adaptor options for Hadoop log files (TaskTracker, DataNode)
Posted by Ariel Rabkin <as...@gmail.com>.
You shouldn't need to recompile hadoop -- you can add Chuwka code to
Hadoop just by putting the chukwa-hadoop-xyz jar in the hadoop lib
directory. (Chukwa also needs json.jar, so you know.) You do need to
reconfigure Hadoop, and chukwa/conf/hadoop-log4j.properties does the
Right Thing.
I'll defer to others for the second half of that question.
On Mon, May 18, 2009 at 7:14 AM, Jiaqi Tan <ta...@gmail.com> wrote:
> Hi Eric,
>
>> Chukwa has a special log4j appender which escapes return character. The
>> multi-lines exception will be stored as a single chunk, and processed as a
>> single chukwa record after Demux.
>
> In this case, I suppose I would need to configure the monitored Hadoop
> cluster to actually use the Chukwa log4j appender? Would I also need
> to recompile the Hadoop of the monitored cluster to include the Chukwa
> code then?
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department
Re: Adaptor options for Hadoop log files (TaskTracker, DataNode)
Posted by Eric Yang <ey...@yahoo-inc.com>.
On 5/18/09 7:14 AM, "Jiaqi Tan" <ta...@gmail.com> wrote:
> Hi Eric,
>
>> Chukwa has a special log4j appender which escapes return character. The
>> multi-lines exception will be stored as a single chunk, and processed as a
>> single chukwa record after Demux.
>
> In this case, I suppose I would need to configure the monitored Hadoop
> cluster to actually use the Chukwa log4j appender? Would I also need
> to recompile the Hadoop of the monitored cluster to include the Chukwa
> code then?
There is no need to recompile Hadoop. Chukwa contains a jar file called
chukwa-hadoop-*-client.jar, and json.jar. Drop those two jar files in lib
directory of the hadoop cluster, and configure log4j.properties in hadoop
conf directory.
>
> Where are these record types defined, and how do they map the the
> processors? Is it a direct <record type name>Processor mapping that's
> automatically done by the Demux?
Record types are defined in the log4j.properties. For example, hadoop has a
appender called DRFA, and the chukwa enabled appender would look like:
log4j.appender.DRFA=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRol
lingFileAppender
log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
log4j.appender.DRFA.recordType=HadoopLog
log4j.appender.DRFA.chukwaClientHostname=localhost
log4j.appender.DRFA.chukwaClientPortNum=9093
The association of HadoopLog record type and the demux class is in
chukwa-demux-conf.xml.
Regards,
Eric
>
>>
>> On 5/17/09 6:42 PM, "Jiaqi Tan" <ta...@gmail.com> wrote:
>>
>>> Hi Ariel,
>>>
>>> So with the CharFileTailingAdaptorUTF8NewLineEscaped, if I have a log
>>> file entry with a multi-line entry, e.g. if there was a Java exception
>>> logged, would each line be separated into a different chunk? If that's
>>> the case, are there any adaptors that would coalesce multi-line log
>>> entries into a single chunk?
>>>
>>> Also, does the data type get resolved by Demux to one of the classes
>>> in org.apache.hadoop.chukwa.extraction.demux.processor.mapper? i.e. if
>>> I wanted to implement my own custom datatype, I should create a Demux
>>> processor and stick it in as one of the classes in that package?
>>>
>>> Thanks,
>>> Jiaqi
>>>
>>> On Sun, May 17, 2009 at 6:19 PM, Ariel Rabkin <as...@gmail.com> wrote:
>>>> It's worth distinguishing two different things.
>>>>
>>>> The adaptor (as in CharFileTailingAdaptorUTF8) is responsible for
>>>> deciding how to break the data into chunks, and how to tag the chunks.
>>>> Probably CharFileTailingAdaptorUTF8NewLineEscaped is right for you.
>>>> (We should really rename that to something shorter!)
>>>>
>>>> The type, like SysLog or NameNodeLog, is stored by the adaptor, and
>>>> passed through as Chunk metadata. It's used to tell the Demux how to
>>>> process that data. The demux-conf has the mapping from datatype to
>>>> processor. For logs, you should be fine just picking a datatype. If
>>>> you aren't using Demux to process the logs, you don't even need to
>>>> write a processor.
>>>>
>>>> --Ari
>>>>
>>>> On Sun, May 17, 2009 at 6:15 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> Which adaptor should I use if I want to process log entries from the
>>>>> TaskTracker and DataNode logs? Should I just use one of the
>>>>> FileTailer adaptors already available (CharFileTailingAdaptorUTF8), or
>>>>> is there a custom type such as the one for SysLog or NameNodeLog when
>>>>> using the CharFileTailingAdaptorUTF8NewLineEscaped adaptor?
>>>>>
>>>>> Is there any documentation available on what the "type" (e.g. SysLog
>>>>> or NameNodeLog) means and how to use it/how it works?
>>>>>
>>>>> Thanks,
>>>>> Jiaqi
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ari Rabkin asrabkin@gmail.com
>>>> UC Berkeley Computer Science Department
>>>>
>>
>>
Re: Adaptor options for Hadoop log files (TaskTracker, DataNode)
Posted by Jiaqi Tan <ta...@gmail.com>.
Hi Eric,
> Chukwa has a special log4j appender which escapes return character. The
> multi-lines exception will be stored as a single chunk, and processed as a
> single chukwa record after Demux.
In this case, I suppose I would need to configure the monitored Hadoop
cluster to actually use the Chukwa log4j appender? Would I also need
to recompile the Hadoop of the monitored cluster to include the Chukwa
code then?
>
> You are on the right track. For your purpose, you may want to create your
> own custom datatype and having a matched chukwa log4j appender record type
> to process the data that you are looking for. To start, you may be
> interested in modifying HadoopLogProcessor, and enhance from there. Chukwa
> is currently streaming all hadoop logs in the same record type (HadoopLog),
> and this part could use some help to carve out the definitions.
Where are these record types defined, and how do they map the the
processors? Is it a direct <record type name>Processor mapping that's
automatically done by the Demux?
Thanks,
Jiaqi
>
> On 5/17/09 6:42 PM, "Jiaqi Tan" <ta...@gmail.com> wrote:
>
>> Hi Ariel,
>>
>> So with the CharFileTailingAdaptorUTF8NewLineEscaped, if I have a log
>> file entry with a multi-line entry, e.g. if there was a Java exception
>> logged, would each line be separated into a different chunk? If that's
>> the case, are there any adaptors that would coalesce multi-line log
>> entries into a single chunk?
>>
>> Also, does the data type get resolved by Demux to one of the classes
>> in org.apache.hadoop.chukwa.extraction.demux.processor.mapper? i.e. if
>> I wanted to implement my own custom datatype, I should create a Demux
>> processor and stick it in as one of the classes in that package?
>>
>> Thanks,
>> Jiaqi
>>
>> On Sun, May 17, 2009 at 6:19 PM, Ariel Rabkin <as...@gmail.com> wrote:
>>> It's worth distinguishing two different things.
>>>
>>> The adaptor (as in CharFileTailingAdaptorUTF8) is responsible for
>>> deciding how to break the data into chunks, and how to tag the chunks.
>>> Probably CharFileTailingAdaptorUTF8NewLineEscaped is right for you.
>>> (We should really rename that to something shorter!)
>>>
>>> The type, like SysLog or NameNodeLog, is stored by the adaptor, and
>>> passed through as Chunk metadata. It's used to tell the Demux how to
>>> process that data. The demux-conf has the mapping from datatype to
>>> processor. For logs, you should be fine just picking a datatype. If
>>> you aren't using Demux to process the logs, you don't even need to
>>> write a processor.
>>>
>>> --Ari
>>>
>>> On Sun, May 17, 2009 at 6:15 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> Which adaptor should I use if I want to process log entries from the
>>>> TaskTracker and DataNode logs? Should I just use one of the
>>>> FileTailer adaptors already available (CharFileTailingAdaptorUTF8), or
>>>> is there a custom type such as the one for SysLog or NameNodeLog when
>>>> using the CharFileTailingAdaptorUTF8NewLineEscaped adaptor?
>>>>
>>>> Is there any documentation available on what the "type" (e.g. SysLog
>>>> or NameNodeLog) means and how to use it/how it works?
>>>>
>>>> Thanks,
>>>> Jiaqi
>>>>
>>>
>>>
>>>
>>> --
>>> Ari Rabkin asrabkin@gmail.com
>>> UC Berkeley Computer Science Department
>>>
>
>
Re: Adaptor options for Hadoop log files (TaskTracker, DataNode)
Posted by Eric Yang <ey...@yahoo-inc.com>.
Hi Jiaqi,
Chukwa has a special log4j appender which escapes return character. The
multi-lines exception will be stored as a single chunk, and processed as a
single chukwa record after Demux.
You are on the right track. For your purpose, you may want to create your
own custom datatype and having a matched chukwa log4j appender record type
to process the data that you are looking for. To start, you may be
interested in modifying HadoopLogProcessor, and enhance from there. Chukwa
is currently streaming all hadoop logs in the same record type (HadoopLog),
and this part could use some help to carve out the definitions. See you
tomorrow.
Regards,
Eric
On 5/17/09 6:42 PM, "Jiaqi Tan" <ta...@gmail.com> wrote:
> Hi Ariel,
>
> So with the CharFileTailingAdaptorUTF8NewLineEscaped, if I have a log
> file entry with a multi-line entry, e.g. if there was a Java exception
> logged, would each line be separated into a different chunk? If that's
> the case, are there any adaptors that would coalesce multi-line log
> entries into a single chunk?
>
> Also, does the data type get resolved by Demux to one of the classes
> in org.apache.hadoop.chukwa.extraction.demux.processor.mapper? i.e. if
> I wanted to implement my own custom datatype, I should create a Demux
> processor and stick it in as one of the classes in that package?
>
> Thanks,
> Jiaqi
>
> On Sun, May 17, 2009 at 6:19 PM, Ariel Rabkin <as...@gmail.com> wrote:
>> It's worth distinguishing two different things.
>>
>> The adaptor (as in CharFileTailingAdaptorUTF8) is responsible for
>> deciding how to break the data into chunks, and how to tag the chunks.
>> Probably CharFileTailingAdaptorUTF8NewLineEscaped is right for you.
>> (We should really rename that to something shorter!)
>>
>> The type, like SysLog or NameNodeLog, is stored by the adaptor, and
>> passed through as Chunk metadata. It's used to tell the Demux how to
>> process that data. The demux-conf has the mapping from datatype to
>> processor. For logs, you should be fine just picking a datatype. If
>> you aren't using Demux to process the logs, you don't even need to
>> write a processor.
>>
>> --Ari
>>
>> On Sun, May 17, 2009 at 6:15 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>>> Hi,
>>>
>>> Which adaptor should I use if I want to process log entries from the
>>> TaskTracker and DataNode logs? Should I just use one of the
>>> FileTailer adaptors already available (CharFileTailingAdaptorUTF8), or
>>> is there a custom type such as the one for SysLog or NameNodeLog when
>>> using the CharFileTailingAdaptorUTF8NewLineEscaped adaptor?
>>>
>>> Is there any documentation available on what the "type" (e.g. SysLog
>>> or NameNodeLog) means and how to use it/how it works?
>>>
>>> Thanks,
>>> Jiaqi
>>>
>>
>>
>>
>> --
>> Ari Rabkin asrabkin@gmail.com
>> UC Berkeley Computer Science Department
>>
Re: Adaptor options for Hadoop log files (TaskTracker, DataNode)
Posted by Jiaqi Tan <ta...@gmail.com>.
Hi Ariel,
So with the CharFileTailingAdaptorUTF8NewLineEscaped, if I have a log
file entry with a multi-line entry, e.g. if there was a Java exception
logged, would each line be separated into a different chunk? If that's
the case, are there any adaptors that would coalesce multi-line log
entries into a single chunk?
Also, does the data type get resolved by Demux to one of the classes
in org.apache.hadoop.chukwa.extraction.demux.processor.mapper? i.e. if
I wanted to implement my own custom datatype, I should create a Demux
processor and stick it in as one of the classes in that package?
Thanks,
Jiaqi
On Sun, May 17, 2009 at 6:19 PM, Ariel Rabkin <as...@gmail.com> wrote:
> It's worth distinguishing two different things.
>
> The adaptor (as in CharFileTailingAdaptorUTF8) is responsible for
> deciding how to break the data into chunks, and how to tag the chunks.
> Probably CharFileTailingAdaptorUTF8NewLineEscaped is right for you.
> (We should really rename that to something shorter!)
>
> The type, like SysLog or NameNodeLog, is stored by the adaptor, and
> passed through as Chunk metadata. It's used to tell the Demux how to
> process that data. The demux-conf has the mapping from datatype to
> processor. For logs, you should be fine just picking a datatype. If
> you aren't using Demux to process the logs, you don't even need to
> write a processor.
>
> --Ari
>
> On Sun, May 17, 2009 at 6:15 PM, Jiaqi Tan <ta...@gmail.com> wrote:
>> Hi,
>>
>> Which adaptor should I use if I want to process log entries from the
>> TaskTracker and DataNode logs? Should I just use one of the
>> FileTailer adaptors already available (CharFileTailingAdaptorUTF8), or
>> is there a custom type such as the one for SysLog or NameNodeLog when
>> using the CharFileTailingAdaptorUTF8NewLineEscaped adaptor?
>>
>> Is there any documentation available on what the "type" (e.g. SysLog
>> or NameNodeLog) means and how to use it/how it works?
>>
>> Thanks,
>> Jiaqi
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>
Re: Adaptor options for Hadoop log files (TaskTracker, DataNode)
Posted by Ariel Rabkin <as...@gmail.com>.
It's worth distinguishing two different things.
The adaptor (as in CharFileTailingAdaptorUTF8) is responsible for
deciding how to break the data into chunks, and how to tag the chunks.
Probably CharFileTailingAdaptorUTF8NewLineEscaped is right for you.
(We should really rename that to something shorter!)
The type, like SysLog or NameNodeLog, is stored by the adaptor, and
passed through as Chunk metadata. It's used to tell the Demux how to
process that data. The demux-conf has the mapping from datatype to
processor. For logs, you should be fine just picking a datatype. If
you aren't using Demux to process the logs, you don't even need to
write a processor.
--Ari
On Sun, May 17, 2009 at 6:15 PM, Jiaqi Tan <ta...@gmail.com> wrote:
> Hi,
>
> Which adaptor should I use if I want to process log entries from the
> TaskTracker and DataNode logs? Should I just use one of the
> FileTailer adaptors already available (CharFileTailingAdaptorUTF8), or
> is there a custom type such as the one for SysLog or NameNodeLog when
> using the CharFileTailingAdaptorUTF8NewLineEscaped adaptor?
>
> Is there any documentation available on what the "type" (e.g. SysLog
> or NameNodeLog) means and how to use it/how it works?
>
> Thanks,
> Jiaqi
>
--
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department