You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Srinivasan Subramanian <ss...@hotmail.com> on 2011/12/26 13:24:34 UTC

Flume test setup with Log4J appender seeing lots of duplicate data

Hi
I just followed the User guide to setup a test flume.  
I have a master I started with: flume master (command line)Collector: flume node_nowatch -n collector (command line)Node: flume node_nowatch (command line)
I can now check the flume master config at localhost:35871.  I can see the collector and agent nodes also connected.
The settings I have for source and sink are as follows:
Collector	default-flow	collectorSource(35853)	collectorSink("file:///var/log/collected/%Y-%m-%d/", "%{customer}-", 3600000)int-logger	default-flow	avroSource(12345)	{split("~", 1, "customer") => agentSink("localhost",35853)}
I have a java test program with the Avro Log4J appender and ran a test application queuing a simple message like "~somename~Test message <count>" in a loop of 20 times.  The flume log agent uses the avrosource and splits based on a regular expression and splits the token somename to a Flume attribute "customer" and sends it further.  The collector sink then writes to a directory  based on date and further on file name based on somename and timestamp.  So far so good as all log messages are collected and processed properly.
However i am seeing huge amounts of duplication of data.  I guess something is wrong with my setup/settings but i am unable to fathom whats wrong.  Any support on getting this right is appreciated.
ThanksSrini