how flume collect different kinds of logs

[zuo ruochen <zu...@qingting.fm>]

Hi ,every one,
    Now, I want to use flume to collect logs from different server, cause that all the servers have different log.I don't know how to classify the logs from collect agent?
    I know i can use different source to collect the log,however if i do this,i should create as many channels as sources to transfer the data to sink.I find that there are selector.header can map the stream ,but the log source is through ’tail -F xxx’ and i don’t know how to set the header?

Re: how flume collect different kinds of logs

[Ahmed Vila <av...@devlogic.eu>]

Hi Zuo,

You haven't described what your classification will be used for.
My guess is that you're collecting from different services and you want
files from the same service (i.e. Tomcat) on different servers to be stored
into a single folder on HDFS.

Also, you haven't explained what transport topology you're aiming for.
Would Syslogd as a collection point would be better idea ?
Are you aiming to store it into the HDFS, HBase or.... ?

Unfortunately, ExecSource doesn't emit any headers, so you can't use
selector.header in that manner.
However, you can use host or static interceptor in order to emit producer's
host and/or static text along with an event so you can later on select it
using selector.header or directly use it in the HDFS sink path.

For the case I've guessed for, I would personally use static interceptor in
order to emit descriptive type of a log (or a service) it's collecting and
in the HDFSSink I would use placeholders in the HDFS path to store them in
different folder in order to simplify configuration and avoid unnecessary
multiple channels and selectors.

On the source side:
agent.sources.src1.interceptors = mylogtypeinterceptor
agent.sources.src1.interceptors.mylogtypeinterceptor.type = static
agent.sources.src1.interceptors.mylogtypeinterceptor.key = mylogtype
agent.sources.src1.interceptors.mylogtypeinterceptor.value = tomcat

On the sink side:
agent.sinks.snk1.hdfs.path = /user/logs/%{mylogtype}



On Fri, Oct 30, 2015 at 9:52 AM, zuo ruochen <zu...@qingting.fm> wrote:

> Hi ,every one,
>     Now, I want to use flume to collect logs from different server, cause
> that all the servers have different log.I don't know how to classify the
> logs from collect agent?
>     I know i can use different source to collect the log,however if i do
> this,i should create as many channels as sources to transfer the data to
> sink.I find that there are selector.header can map the stream ,but the log
> source is through ’tail -F xxx’ and i don’t know how to set the header?
>
>


-- 

Best regards,
Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina

Office : +387 33 942 123
Mobile: +387 62 139 348

Website: www.devlogic.eu
E-mail   : avila@devlogic.eu
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended
recipient(s) only. This email contains confidential information. It should
not be copied, disclosed to, retained or used by, any party other than the
intended recipient. Any unauthorised distribution, dissemination or copying
of this E-mail or its attachments, and/or any use of any information
contained in them, is strictly prohibited and may be illegal. If you are
not an intended recipient then please promptly delete this e-mail and any
attachment and all copies and inform the sender directly via email. Any
emails that you send to us may be monitored by systems or persons other
than the named communicant for the purposes of ascertaining whether the
communication complies with the law and company policies.

-- 
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended 
recipient(s) only. This email contains confidential information. It should 
not be copied, disclosed to, retained or used by, any party other than the 
intended recipient. Any unauthorised distribution, dissemination or copying 
of this E-mail or its attachments, and/or any use of any information 
contained in them, is strictly prohibited and may be illegal. If you are 
not an intended recipient then please promptly delete this e-mail and any 
attachment and all copies and inform the sender directly via email. Any 
emails that you send to us may be monitored by systems or persons other 
than the named communicant for the purposes of ascertaining whether the 
communication complies with the law and company policies.