You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by lancexxx <fe...@gmail.com> on 2012/11/02 04:16:06 UTC
hdfs sink output mojibake
I got a error seems mojibake.
this is my flume.conf,is it any wrong?
---------
agent.sources = source1
agent.channels = channel1
agent.sinks = sink1
agent.channels.channel1.type = memory
agent.channels.channel1.capacity = 100000
agent.channels.channel1.transactionCapacity = 8000
agent.channels.channel1.keep-alive = 3
agent.sources.source1.type = avro
agent.sources.source1.bind = 0.0.0.0
agent.sources.source1.port = 4141
agent.sources.source1.channels = channel1
agent.sinks.sink1.type = hdfs
agent.sinks.sink1.hdfs.path = hdfs://flume:9000/g
agent.sinks.sink1.hdfs.fileType = DataStream
agent.sinks.sink1.hdfs.rollInterval = 300
agent.sinks.sink1.hdfs.rollSize = 0
agent.sinks.sink1.hdfs.rollCount = 0
agent.sinks.sink1.hdfs.batchSize = 1000
agent.sinks.sink1.serializer = AVRO_EVENT
agent.sinks.sink1.serializer.compressionCodec = snappy
agent.sinks.sink1.channel = channel1
-------------
--
lancexxx
Re: hdfs sink output mojibake
Posted by lancexxx <fe...@gmail.com>.
the logs of flume with no error,flume-ng looks like completely normal except the web logs what collecting and output by flume-ng on hdfs is messy code 。
please look at the e-mail attachment.
--
lancexxx
On 2012年11月3日Saturday at 下午3:06, Alexander Lorenz wrote:
> Please attach the logs or console debug messages. What's exactly the issue?
>
> - alex
>
> On Nov 2, 2012, at 4:16 AM, lancexxx <fervent.linux@gmail.com (mailto:fervent.linux@gmail.com)> wrote:
>
> > I got a error seems mojibake.
> >
> >
> >
> > this is my flume.conf,is it any wrong?
> > ---------
> > agent.sources = source1
> > agent.channels = channel1
> > agent.sinks = sink1
> >
> > agent.channels.channel1.type = memory
> > agent.channels.channel1.capacity = 100000
> > agent.channels.channel1.transactionCapacity = 8000
> > agent.channels.channel1.keep-alive = 3
> >
> > agent.sources.source1.type = avro
> > agent.sources.source1.bind = 0.0.0.0
> > agent.sources.source1.port = 4141
> > agent.sources.source1.channels = channel1
> >
> >
> >
> > agent.sinks.sink1.type = hdfs
> > agent.sinks.sink1.hdfs.path = hdfs://flume:9000/g
> > agent.sinks.sink1.hdfs.fileType = DataStream
> > agent.sinks.sink1.hdfs.rollInterval = 300
> > agent.sinks.sink1.hdfs.rollSize = 0
> > agent.sinks.sink1.hdfs.rollCount = 0
> > agent.sinks.sink1.hdfs.batchSize = 1000
> > agent.sinks.sink1.serializer = AVRO_EVENT
> > agent.sinks.sink1.serializer.compressionCodec = snappy
> > agent.sinks.sink1.channel = channel1
> > -------------
> >
> >
> > --
> > lancexxx
> >
> > <_g_FlumeData.1351825007314.tmp>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>
Re: hdfs sink output mojibake
Posted by Alexander Lorenz <wg...@gmail.com>.
Please attach the logs or console debug messages. What's exactly the issue?
- alex
On Nov 2, 2012, at 4:16 AM, lancexxx <fe...@gmail.com> wrote:
> I got a error seems mojibake.
>
>
>
> this is my flume.conf,is it any wrong?
> ---------
> agent.sources = source1
> agent.channels = channel1
> agent.sinks = sink1
>
> agent.channels.channel1.type = memory
> agent.channels.channel1.capacity = 100000
> agent.channels.channel1.transactionCapacity = 8000
> agent.channels.channel1.keep-alive = 3
>
> agent.sources.source1.type = avro
> agent.sources.source1.bind = 0.0.0.0
> agent.sources.source1.port = 4141
> agent.sources.source1.channels = channel1
>
>
>
> agent.sinks.sink1.type = hdfs
> agent.sinks.sink1.hdfs.path = hdfs://flume:9000/g
> agent.sinks.sink1.hdfs.fileType = DataStream
> agent.sinks.sink1.hdfs.rollInterval = 300
> agent.sinks.sink1.hdfs.rollSize = 0
> agent.sinks.sink1.hdfs.rollCount = 0
> agent.sinks.sink1.hdfs.batchSize = 1000
> agent.sinks.sink1.serializer = AVRO_EVENT
> agent.sinks.sink1.serializer.compressionCodec = snappy
> agent.sinks.sink1.channel = channel1
> -------------
>
>
> --
> lancexxx
>
> <_g_FlumeData.1351825007314.tmp>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF