You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by Apache Wiki <wi...@apache.org> on 2009/06/18 23:42:18 UTC
[Hadoop Wiki] Update of "Chukwa Adaptors List" by AriRabkin
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The following page has been changed by AriRabkin:
http://wiki.apache.org/hadoop/Chukwa_Adaptors_List
The comment on the change is:
document adaptors more clearly.
------------------------------------------------------------------------------
- The following adaptors are built into Chukwa and can be added to the agent using the command line provided after each. These lines can also be added to the conf/initial_adaptors file to have agents run adaptors at startup time.
+ In Chukwa, data is collected by adaptors. Adaptors run inside the agent process, and can be dynamically started and stopped.
- In order to start an adaptor on an agent, you can telnet into the agent (i.e. and give the "ADD" command line to the agent.
+ (mention that commands can go in initial_adaptors or else at control port)
- '''!FileTailer'''
+ The general syntax of the add command is as follows:
- '''!FileTailingAdaptor'''
+ {{{add [name =] <adaptor_class_name> <datatype> <adaptor specific params> <initial offset>.}}}
+
+ The `name` field lets you give this adaptor a meaningful name, in order to stop it later. If you do not specify a name, Chukwa will autogenerate one by hashing the class name and parameters. The adaptor class name specifies what adaptor to run. The `datatype` specifies how to process the collected data. The initial offset should normally be zero; specifying a nonzero initial offset lets you do advanced things like only processing a portion of a file.
- '''CharFileTailingAdaptorUTF8'''
+ The adaptors built into Chukwa are as follows:
- '''CharFileTailingAdaptorUT!F8NewLineEscaped''' add org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.CharFileTailingAdaptorUT!F8NewLineEscaped SysLog 0 /var/log/messages 0
+ '''!FileAdaptor''': Push a whole file, as one Chunk, then exit. Requires file name as parameter.
- '''!TerminatorThread'''
+ Example: {{{add FileTailer }}}
+ '''!filetailer.FileTailingAdaptor''': The
+ Pushes the file as flat bytes, ignoring the content. Useful for streaming binary data.
+ '''!filetailer.CharFileTailingAdaptorUTF8'''
+ The same, but breaking chunks only at carriage returns. Useful for most ASCII log file formats.
+ '''!filetailer.CharFileTailingAdaptorUT!F8NewLineEscaped'''
+ The same, but breaking chunks only at non-escaped carriage returns. Useful for pushing Chukwa-formatted log files, where exception stack traces stay in a single chunk.
+