You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by yogi nerella <yn...@gmail.com> on 2013/01/23 23:36:52 UTC
flume to collect log files from various sources
Hi,
I am a new bee for flume, would like to get some feedback on if what I am
doing works correctly.
My application runs on multiple hosts, I want to collect all log files to a
central location.
1. In My application I will ship all relevant log4jappender jar files, and
configure the agent's host, port information.
2. I will run a simple agent, and configure a source (avro), channel
(memory), and sink (file_roll).
File_roll will write the events recieved via AVRO source and write them to
the corresponding file_roll sink.
3. Do I need one agent for each host, or is there a way I can configure to
write to multiple files, dynamically based on the real source the message
is getting recieved.
4. If I modify the configuration file, will the agent re-read the
information. (Ex: I want to add a new host, and collect the log files to
a new directory).
5. In my simple test, messages sent from my app are getting lost, and I
see the following information in the console.
Classpath:
C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize: 0;
Using default value.
14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
{tcpNoDelay=true, connectTimeoutMillis=20000}
14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /16.90.218.66:44444
14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
log4j:ERROR Flume append() failed.
14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST 2013
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
Appreciate your help
Yogi
Re: flume to collect log files from various sources
Posted by Alexander Alten-Lorenz <wg...@gmail.com>.
He's using flume-1.3.1, not 0.x:
C:\apache\apache-flume-1.3.1-bin
Yes, flume runs on Windows, just compile it.
@ Yogi:
15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
log4j:ERROR Flume append() failed.
Can you connect per Telnet to the ports you have defined? What says netstat (or a similar tool on Windows to show open ports). Firewall disabled?
- Alex
On Jan 29, 2013, at 4:40 AM, Sri Ramya <ra...@gmail.com> wrote:
> In side flume there is a folder conf/. in that u will find the
> flume_site.xml
> and bin/ you find binaries. download flume from this link
> http://archive.cloudera.com/cdh/3/flume-0.9.4-cdh3u4.tar.gz
>
> and follow the flume user guide. u will able to connect .
>
>
> On Mon, Jan 28, 2013 at 9:59 PM, yogi nerella <yn...@gmail.com> wrote:
>
>> Hello Ramya,
>>
>> The above link shows example commands to configure master, node etc.,
>> But I donot see any "flume" binary? or "config" binary?
>> or flume_site.xml file in OOB flume-1.3.1-bin or flume-1.4.0-bin.
>>
>>
>> Am I missing something?
>> How do I set my agent's as nodes, and my collector's as masters etc?
>>
>>
>> Thanks,
>> Yogi
>>
>>
>>
>>
>> On Sun, Jan 27, 2013 at 10:35 PM, Sri Ramya <ra...@gmail.com> wrote:
>>
>>> sorry for late reply
>>> http://archive.cloudera.com/cdh/3/flume/UserGuide/
>>> check out this u will get all details about how to configure. And i never
>>> tried log4jappender.
>>>
>>>
>>> On Thu, Jan 24, 2013 at 10:18 PM, yogi nerella <yn...@gmail.com>wrote:
>>>
>>>> Hello Ramya,
>>>>
>>>> Flume agent have to be installed on every host?
>>>> I want to avoid this by integrating log4jappender into my app, and
>>>> to remove one additional overhead for administrators.
>>>> Is this a problem?
>>>>
>>>>
>>>> Collecting data using "taildir" command?
>>>> 1. If I have one flume agent sending log files from one host to
>>>> another agent who is listening.
>>>> 2. How do I configure the receiving agent such that, it can
>>>> separate the log files again and write as it recieved.
>>>>
>>>>
>>>> Any sample configuration files for either 'taildir' or 'multitail'
>>>> commands?
>>>>
>>>> Issue with log4jappender:
>>>>
>>>> I am evaluating log4jappender, and using this I am not able to send
>>>> events to the flume agent.
>>>>
>>>> When I turn on debugging all the log messages I see are as follows.
>>>>
>>>> Classpath:
>>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
>>>> 0; Using default value.
>>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
>>>> {tcpNoDelay=true, connectTimeoutMillis=20000}
>>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>>> 16.90.218.66:44444
>>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>>> log4j:ERROR Flume append() failed.
>>>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>>>> 2013
>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>>
>>>>
>>>> Yogi
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>>
>>>>> there is an error displayed on console Flume append() failed. Try to
>>>>> resolve .
>>>>> explain me what is your exact problem.
>>>>>
>>>>> Note:
>>>>> 1, Flume agent have to install in every host from where u want to
>>>>> collect data.
>>>>> 2, If u want to collect data from a directory then you have to to use
>>>>> 'taildir' command.
>>>>> 3, If u want to collect data from different files u have to use
>>>>> 'multitail' command.
>>>>>
>>>>> I think this info will be use full for u.
>>>>> thank you
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <yn...@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am a new bee for flume, would like to get some feedback on if what I
>>>>>> am doing works correctly.
>>>>>>
>>>>>>
>>>>>> My application runs on multiple hosts, I want to collect all log files
>>>>>> to a central location.
>>>>>>
>>>>>> 1. In My application I will ship all relevant log4jappender jar
>>>>>> files, and configure the agent's host, port information.
>>>>>>
>>>>>> 2. I will run a simple agent, and configure a source (avro), channel
>>>>>> (memory), and sink (file_roll).
>>>>>>
>>>>>> File_roll will write the events recieved via AVRO source and write
>>>>>> them to the corresponding file_roll sink.
>>>>>>
>>>>>> 3. Do I need one agent for each host, or is there a way I can
>>>>>> configure to write to multiple files, dynamically based on the real source
>>>>>> the message is getting recieved.
>>>>>>
>>>>>> 4. If I modify the configuration file, will the agent re-read the
>>>>>> information. (Ex: I want to add a new host, and collect the log files to
>>>>>> a new directory).
>>>>>>
>>>>>> 5. In my simple test, messages sent from my app are getting lost, and
>>>>>> I see the following information in the console.
>>>>>>
>>>>>>
>>>>>> Classpath:
>>>>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>>>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>>>>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for
>>>>>> batchSize: 0; Using default value.
>>>>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap
>>>>>> options: {tcpNoDelay=true, connectTimeoutMillis=20000}
>>>>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>>>>> 16.90.218.66:44444
>>>>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>>>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>>>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>>>>> log4j:ERROR Flume append() failed.
>>>>>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29
>>>>>> PST 2013
>>>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>>
>>>>>>
>>>>>> Appreciate your help
>>>>>> Yogi
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: flume to collect log files from various sources
Posted by Sri Ramya <ra...@gmail.com>.
In side flume there is a folder conf/. in that u will find the
flume_site.xml
and bin/ you find binaries. download flume from this link
http://archive.cloudera.com/cdh/3/flume-0.9.4-cdh3u4.tar.gz
and follow the flume user guide. u will able to connect .
On Mon, Jan 28, 2013 at 9:59 PM, yogi nerella <yn...@gmail.com> wrote:
> Hello Ramya,
>
> The above link shows example commands to configure master, node etc.,
> But I donot see any "flume" binary? or "config" binary?
> or flume_site.xml file in OOB flume-1.3.1-bin or flume-1.4.0-bin.
>
>
> Am I missing something?
> How do I set my agent's as nodes, and my collector's as masters etc?
>
>
> Thanks,
> Yogi
>
>
>
>
> On Sun, Jan 27, 2013 at 10:35 PM, Sri Ramya <ra...@gmail.com> wrote:
>
>> sorry for late reply
>> http://archive.cloudera.com/cdh/3/flume/UserGuide/
>> check out this u will get all details about how to configure. And i never
>> tried log4jappender.
>>
>>
>> On Thu, Jan 24, 2013 at 10:18 PM, yogi nerella <yn...@gmail.com>wrote:
>>
>>> Hello Ramya,
>>>
>>> Flume agent have to be installed on every host?
>>> I want to avoid this by integrating log4jappender into my app, and
>>> to remove one additional overhead for administrators.
>>> Is this a problem?
>>>
>>>
>>> Collecting data using "taildir" command?
>>> 1. If I have one flume agent sending log files from one host to
>>> another agent who is listening.
>>> 2. How do I configure the receiving agent such that, it can
>>> separate the log files again and write as it recieved.
>>>
>>>
>>> Any sample configuration files for either 'taildir' or 'multitail'
>>> commands?
>>>
>>> Issue with log4jappender:
>>>
>>> I am evaluating log4jappender, and using this I am not able to send
>>> events to the flume agent.
>>>
>>> When I turn on debugging all the log messages I see are as follows.
>>>
>>> Classpath:
>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
>>> 0; Using default value.
>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
>>> {tcpNoDelay=true, connectTimeoutMillis=20000}
>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>> 16.90.218.66:44444
>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>> log4j:ERROR Flume append() failed.
>>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>>> 2013
>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>>
>>>
>>> Yogi
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <ra...@gmail.com> wrote:
>>>
>>>> there is an error displayed on console Flume append() failed. Try to
>>>> resolve .
>>>> explain me what is your exact problem.
>>>>
>>>> Note:
>>>> 1, Flume agent have to install in every host from where u want to
>>>> collect data.
>>>> 2, If u want to collect data from a directory then you have to to use
>>>> 'taildir' command.
>>>> 3, If u want to collect data from different files u have to use
>>>> 'multitail' command.
>>>>
>>>> I think this info will be use full for u.
>>>> thank you
>>>>
>>>>
>>>>
>>>> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <yn...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am a new bee for flume, would like to get some feedback on if what I
>>>>> am doing works correctly.
>>>>>
>>>>>
>>>>> My application runs on multiple hosts, I want to collect all log files
>>>>> to a central location.
>>>>>
>>>>> 1. In My application I will ship all relevant log4jappender jar
>>>>> files, and configure the agent's host, port information.
>>>>>
>>>>> 2. I will run a simple agent, and configure a source (avro), channel
>>>>> (memory), and sink (file_roll).
>>>>>
>>>>> File_roll will write the events recieved via AVRO source and write
>>>>> them to the corresponding file_roll sink.
>>>>>
>>>>> 3. Do I need one agent for each host, or is there a way I can
>>>>> configure to write to multiple files, dynamically based on the real source
>>>>> the message is getting recieved.
>>>>>
>>>>> 4. If I modify the configuration file, will the agent re-read the
>>>>> information. (Ex: I want to add a new host, and collect the log files to
>>>>> a new directory).
>>>>>
>>>>> 5. In my simple test, messages sent from my app are getting lost, and
>>>>> I see the following information in the console.
>>>>>
>>>>>
>>>>> Classpath:
>>>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>>>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for
>>>>> batchSize: 0; Using default value.
>>>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap
>>>>> options: {tcpNoDelay=true, connectTimeoutMillis=20000}
>>>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>>>> 16.90.218.66:44444
>>>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>>>> log4j:ERROR Flume append() failed.
>>>>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29
>>>>> PST 2013
>>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>> 16.90.218.66/16.90.218.66:44444
>>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>> 16.90.218.66/16.90.218.66:44444
>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>> 16.90.218.66/16.90.218.66:44444
>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>> 16.90.218.66/16.90.218.66:44444
>>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>>> 16.90.218.66/16.90.218.66:44444
>>>>>
>>>>>
>>>>> Appreciate your help
>>>>> Yogi
>>>>>
>>>>
>>>>
>>>
>>
>
Flume NG docs on duplicate or dropped log events
Posted by Friso van Vollenhoven <fv...@xebia.com>.
Hi All,
Is there any documentation on the circumstances under which flume ng will either drop events or possibly send events twice resulting in duplicates?
I seem to be able to run into both situations with a test setup under high contention, using a agent1[syslog source --> file channel --> avro sink] --> agent2[avro source, file channel, hdfs sink]. I drop events with the default values for the timeouts on the file channels in combination with letting agent1 become unavailable for some period of time (causing rsyslog to build up a queue). The same situation with higher timeouts leads to a number of duplicate events (about 500 after 2.5M events).
(BTW: is there an official ascii art notation for flume setups?)
Thanks for any pointers,
Friso
Re: flume to collect log files from various sources
Posted by yogi nerella <yn...@gmail.com>.
Hello Ramya,
The above link shows example commands to configure master, node etc.,
But I donot see any "flume" binary? or "config" binary?
or flume_site.xml file in OOB flume-1.3.1-bin or flume-1.4.0-bin.
Am I missing something?
How do I set my agent's as nodes, and my collector's as masters etc?
Thanks,
Yogi
On Sun, Jan 27, 2013 at 10:35 PM, Sri Ramya <ra...@gmail.com> wrote:
> sorry for late reply
> http://archive.cloudera.com/cdh/3/flume/UserGuide/
> check out this u will get all details about how to configure. And i never
> tried log4jappender.
>
>
> On Thu, Jan 24, 2013 at 10:18 PM, yogi nerella <yn...@gmail.com>wrote:
>
>> Hello Ramya,
>>
>> Flume agent have to be installed on every host?
>> I want to avoid this by integrating log4jappender into my app, and to
>> remove one additional overhead for administrators.
>> Is this a problem?
>>
>>
>> Collecting data using "taildir" command?
>> 1. If I have one flume agent sending log files from one host to
>> another agent who is listening.
>> 2. How do I configure the receiving agent such that, it can
>> separate the log files again and write as it recieved.
>>
>>
>> Any sample configuration files for either 'taildir' or 'multitail'
>> commands?
>>
>> Issue with log4jappender:
>>
>> I am evaluating log4jappender, and using this I am not able to send
>> events to the flume agent.
>>
>> When I turn on debugging all the log messages I see are as follows.
>>
>> Classpath:
>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
>> 0; Using default value.
>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
>> {tcpNoDelay=true, connectTimeoutMillis=20000}
>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>> 16.90.218.66:44444
>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>> log4j:ERROR Flume append() failed.
>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>> 2013
>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>>
>>
>> Yogi
>>
>>
>>
>>
>>
>> On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <ra...@gmail.com> wrote:
>>
>>> there is an error displayed on console Flume append() failed. Try to
>>> resolve .
>>> explain me what is your exact problem.
>>>
>>> Note:
>>> 1, Flume agent have to install in every host from where u want to
>>> collect data.
>>> 2, If u want to collect data from a directory then you have to to use
>>> 'taildir' command.
>>> 3, If u want to collect data from different files u have to use
>>> 'multitail' command.
>>>
>>> I think this info will be use full for u.
>>> thank you
>>>
>>>
>>>
>>> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <yn...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I am a new bee for flume, would like to get some feedback on if what I
>>>> am doing works correctly.
>>>>
>>>>
>>>> My application runs on multiple hosts, I want to collect all log files
>>>> to a central location.
>>>>
>>>> 1. In My application I will ship all relevant log4jappender jar files,
>>>> and configure the agent's host, port information.
>>>>
>>>> 2. I will run a simple agent, and configure a source (avro), channel
>>>> (memory), and sink (file_roll).
>>>>
>>>> File_roll will write the events recieved via AVRO source and write them
>>>> to the corresponding file_roll sink.
>>>>
>>>> 3. Do I need one agent for each host, or is there a way I can configure
>>>> to write to multiple files, dynamically based on the real source the
>>>> message is getting recieved.
>>>>
>>>> 4. If I modify the configuration file, will the agent re-read the
>>>> information. (Ex: I want to add a new host, and collect the log files to
>>>> a new directory).
>>>>
>>>> 5. In my simple test, messages sent from my app are getting lost, and
>>>> I see the following information in the console.
>>>>
>>>>
>>>> Classpath:
>>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for
>>>> batchSize: 0; Using default value.
>>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap
>>>> options: {tcpNoDelay=true, connectTimeoutMillis=20000}
>>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>>> 16.90.218.66:44444
>>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>>> log4j:ERROR Flume append() failed.
>>>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>>>> 2013
>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>>> 16.90.218.66/16.90.218.66:44444
>>>>
>>>>
>>>> Appreciate your help
>>>> Yogi
>>>>
>>>
>>>
>>
>
Re: flume to collect log files from various sources
Posted by Sri Ramya <ra...@gmail.com>.
sorry for late reply
http://archive.cloudera.com/cdh/3/flume/UserGuide/
check out this u will get all details about how to configure. And i never
tried log4jappender.
On Thu, Jan 24, 2013 at 10:18 PM, yogi nerella <yn...@gmail.com>wrote:
> Hello Ramya,
>
> Flume agent have to be installed on every host?
> I want to avoid this by integrating log4jappender into my app, and to
> remove one additional overhead for administrators.
> Is this a problem?
>
>
> Collecting data using "taildir" command?
> 1. If I have one flume agent sending log files from one host to
> another agent who is listening.
> 2. How do I configure the receiving agent such that, it can separate
> the log files again and write as it recieved.
>
>
> Any sample configuration files for either 'taildir' or 'multitail'
> commands?
>
> Issue with log4jappender:
>
> I am evaluating log4jappender, and using this I am not able to send events
> to the flume agent.
>
> When I turn on debugging all the log messages I see are as follows.
>
> Classpath:
> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
> 0; Using default value.
> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
> {tcpNoDelay=true, connectTimeoutMillis=20000}
> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
> 16.90.218.66:44444
> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
> log4j:ERROR Flume append() failed.
> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
> 2013
> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
>
>
> Yogi
>
>
>
>
>
> On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <ra...@gmail.com> wrote:
>
>> there is an error displayed on console Flume append() failed. Try to
>> resolve .
>> explain me what is your exact problem.
>>
>> Note:
>> 1, Flume agent have to install in every host from where u want to collect
>> data.
>> 2, If u want to collect data from a directory then you have to to use
>> 'taildir' command.
>> 3, If u want to collect data from different files u have to use
>> 'multitail' command.
>>
>> I think this info will be use full for u.
>> thank you
>>
>>
>>
>> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <yn...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I am a new bee for flume, would like to get some feedback on if what I
>>> am doing works correctly.
>>>
>>>
>>> My application runs on multiple hosts, I want to collect all log files
>>> to a central location.
>>>
>>> 1. In My application I will ship all relevant log4jappender jar files,
>>> and configure the agent's host, port information.
>>>
>>> 2. I will run a simple agent, and configure a source (avro), channel
>>> (memory), and sink (file_roll).
>>>
>>> File_roll will write the events recieved via AVRO source and write them
>>> to the corresponding file_roll sink.
>>>
>>> 3. Do I need one agent for each host, or is there a way I can configure
>>> to write to multiple files, dynamically based on the real source the
>>> message is getting recieved.
>>>
>>> 4. If I modify the configuration file, will the agent re-read the
>>> information. (Ex: I want to add a new host, and collect the log files to
>>> a new directory).
>>>
>>> 5. In my simple test, messages sent from my app are getting lost, and I
>>> see the following information in the console.
>>>
>>>
>>> Classpath:
>>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
>>> 0; Using default value.
>>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
>>> {tcpNoDelay=true, connectTimeoutMillis=20000}
>>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>>> 16.90.218.66:44444
>>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>>> log4j:ERROR Flume append() failed.
>>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>>> 2013
>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>>> 16.90.218.66/16.90.218.66:44444
>>>
>>>
>>> Appreciate your help
>>> Yogi
>>>
>>
>>
>
Re: flume to collect log files from various sources
Posted by yogi nerella <yn...@gmail.com>.
Hello Ramya,
Flume agent have to be installed on every host?
I want to avoid this by integrating log4jappender into my app, and to
remove one additional overhead for administrators.
Is this a problem?
Collecting data using "taildir" command?
1. If I have one flume agent sending log files from one host to
another agent who is listening.
2. How do I configure the receiving agent such that, it can separate
the log files again and write as it recieved.
Any sample configuration files for either 'taildir' or 'multitail'
commands?
Issue with log4jappender:
I am evaluating log4jappender, and using this I am not able to send events
to the flume agent.
When I turn on debugging all the log messages I see are as follows.
Classpath:
C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize: 0;
Using default value.
14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
{tcpNoDelay=true, connectTimeoutMillis=20000}
14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /16.90.218.66:44444
14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
log4j:ERROR Flume append() failed.
14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST 2013
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
16.90.218.66/16.90.218.66:44444
Yogi
On Wed, Jan 23, 2013 at 8:57 PM, Sri Ramya <ra...@gmail.com> wrote:
> there is an error displayed on console Flume append() failed. Try to
> resolve .
> explain me what is your exact problem.
>
> Note:
> 1, Flume agent have to install in every host from where u want to collect
> data.
> 2, If u want to collect data from a directory then you have to to use
> 'taildir' command.
> 3, If u want to collect data from different files u have to use
> 'multitail' command.
>
> I think this info will be use full for u.
> thank you
>
>
>
> On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <yn...@gmail.com>wrote:
>
>> Hi,
>>
>> I am a new bee for flume, would like to get some feedback on if what I am
>> doing works correctly.
>>
>>
>> My application runs on multiple hosts, I want to collect all log files to
>> a central location.
>>
>> 1. In My application I will ship all relevant log4jappender jar files,
>> and configure the agent's host, port information.
>>
>> 2. I will run a simple agent, and configure a source (avro), channel
>> (memory), and sink (file_roll).
>>
>> File_roll will write the events recieved via AVRO source and write them
>> to the corresponding file_roll sink.
>>
>> 3. Do I need one agent for each host, or is there a way I can configure
>> to write to multiple files, dynamically based on the real source the
>> message is getting recieved.
>>
>> 4. If I modify the configuration file, will the agent re-read the
>> information. (Ex: I want to add a new host, and collect the log files to
>> a new directory).
>>
>> 5. In my simple test, messages sent from my app are getting lost, and I
>> see the following information in the console.
>>
>>
>> Classpath:
>> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
>> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
>> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
>> 0; Using default value.
>> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
>> {tcpNoDelay=true, connectTimeoutMillis=20000}
>> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
>> 16.90.218.66:44444
>> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
>> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
>> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
>> log4j:ERROR Flume append() failed.
>> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
>> 2013
>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
>> 16.90.218.66/16.90.218.66:44444
>>
>>
>> Appreciate your help
>> Yogi
>>
>
>
Re: flume to collect log files from various sources
Posted by Sri Ramya <ra...@gmail.com>.
there is an error displayed on console Flume append() failed. Try to
resolve .
explain me what is your exact problem.
Note:
1, Flume agent have to install in every host from where u want to collect
data.
2, If u want to collect data from a directory then you have to to use
'taildir' command.
3, If u want to collect data from different files u have to use 'multitail'
command.
I think this info will be use full for u.
thank you
On Thu, Jan 24, 2013 at 4:06 AM, yogi nerella <yn...@gmail.com> wrote:
> Hi,
>
> I am a new bee for flume, would like to get some feedback on if what I am
> doing works correctly.
>
>
> My application runs on multiple hosts, I want to collect all log files to
> a central location.
>
> 1. In My application I will ship all relevant log4jappender jar files,
> and configure the agent's host, port information.
>
> 2. I will run a simple agent, and configure a source (avro), channel
> (memory), and sink (file_roll).
>
> File_roll will write the events recieved via AVRO source and write them to
> the corresponding file_roll sink.
>
> 3. Do I need one agent for each host, or is there a way I can configure to
> write to multiple files, dynamically based on the real source the message
> is getting recieved.
>
> 4. If I modify the configuration file, will the agent re-read the
> information. (Ex: I want to add a new host, and collect the log files to
> a new directory).
>
> 5. In my simple test, messages sent from my app are getting lost, and I
> see the following information in the console.
>
>
> Classpath:
> C:\ServiceManager\workspace\mongodb\Mongodb\bin;C:\apache\apache-flume-1.3.1-bin\lib\avro-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\avro-ipc-1.7.2.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-core-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-log4jappender-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\flume-ng-sdk-1.3.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-core-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\jackson-mapper-asl-1.9.3.jar;C:\apache\apache-flume-1.3.1-bin\lib\netty-3.4.0.Final.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-api-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\slf4j-log4j12-1.6.1.jar;C:\apache\apache-flume-1.3.1-bin\lib\log4j-1.2.16.jar
> 14:27:29,600 DEBUG NettyAvroRpcClient:420 - Batch size string = 0
> 14:27:29,604 WARN NettyAvroRpcClient:426 - Invalid value for batchSize:
> 0; Using default value.
> 14:27:29,649 DEBUG NettyTransceiver:195 - Using Netty bootstrap options:
> {tcpNoDelay=true, connectTimeoutMillis=20000}
> 14:27:29,649 DEBUG NettyTransceiver:252 - Connecting to /
> 16.90.218.66:44444
> 14:27:29,675 DEBUG NettyTransceiver:491 - [id: 0x01480773] OPEN
> 14:27:29,699 DEBUG NettyTransceiver:491 - [id: 0x01480773, /
> 15.80.67.94:61452 => /16.90.218.66:44444] BOUND: /15.80.67.94:61452
> log4j:ERROR Flume append() failed.
> 14:27:29,940 INFO LogTest:13 - main started at Wed Jan 23 14:27:29 PST
> 2013
> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,501 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
> 14:27:54,502 DEBUG NettyTransceiver:314 - Disconnecting from
> 16.90.218.66/16.90.218.66:44444
>
>
> Appreciate your help
> Yogi
>