You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Amit Handa <am...@gmail.com> on 2012/07/04 09:36:04 UTC
FlumeNG Error while going with Avro (Source) and HDFS (Sink)
Hi,
I am exploring Flume-NG,
1) While going for Avro (as Source) , and HDFS as Sink. I am getting error
"[ERROR -
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:125)]
RPC connection error :" (error log file attached). Kindly help in resolving
it
1 a) flume.conf file is as follows
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 107.101.199.29
agent1.sources.avro-source1.port = 41414
# Define a hdfs sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.HDFS.channel = ch1
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://
107.101.199.29:54310/user/hadoop-node1/flume-test/
agent1.sinks.HDFS.hdfs.file.Type = DataStream
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = HDFS
1 b) Mine is single node hadoop setup. command executed for flume are
bin/flume-ng agent --conf ./conf/ -f conf/flumeAVRO_HDFS.conf
-n agent1 (in one console)
bin/flume-ng avro-client --conf conf -H 107.108.199.29 -p 41414
-F /home/hadoop-node1/Desktop/my.txt (in other console for Avro CLient) [
also tried command with -H localhost option), but same error.
1 c) hadoop version 0.20 is used
2) Provide me links how to explore various combination of source channel,
and sink w.r.t Flume-NG
With Regards,
Amit
Re: FlumeNG Error while going with Avro (Source) and HDFS (Sink)
Posted by Mike Percy <mp...@cloudera.com>.
On Wed, Jul 4, 2012 at 12:36 AM, Amit Handa <am...@gmail.com> wrote:
>
> 1 c) hadoop version 0.20 is used
>
Because of the requirement for sync() in the HDFS sink, you must run an
appropriate version of Hadoop (HBase has the same requirement). Hadoop
0.20.2 will not work. Minimum compatible versions of Hadoop include:
branch-append, 0.20.205+, Hadoop 1.0.0+, and CDH3 / CDH4.
Regards,
Mike
Re: FlumeNG Error while going with Avro (Source) and HDFS
(Sink)
Posted by Hari Shreedharan <hs...@cloudera.com>.
Are you behind a NAT? If you are, then you need to set up port forwarding on your NAT device. An easy way to check is to start an nc server: nc -l <port_number>
and then try connecting to it using: nc 107.108.199.29 <port_number>.
Thanks
Hari
--
Hari Shreedharan
On Wednesday, July 4, 2012 at 5:22 AM, Amit Handa wrote:
> Hi,
>
> 1) While going for Avro (as Source) , and HDFS as Sink. I am getting error "[ERROR - org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:125)] RPC connection error :". Kindly help in resolving it
>
> 1 a) flume.conf file is as follows
>
> # Define a memory channel called ch1 on agent1
> agent1.channels.ch1.type = memory
>
> # Define an Avro source called avro-source1 on agent1 and tell it
> # to bind to 0.0.0.0:41414 (http://0.0.0.0:41414). Connect it to channel ch1.
> agent1.sources.avro-source1.channels = ch1
> agent1.sources.avro-source1.type = avro
> agent1.sources.avro-source1.bind = 107.101.199.29
> agent1.sources.avro-source1.port = 41414
>
> # Define a hdfs sink that simply logs all events it receives
> # and connect it to the other end of the same channel.
> agent1.sinks.HDFS.channel = ch1
> agent1.sinks.HDFS.type = hdfs
> agent1.sinks.HDFS.hdfs.path = hdfs://107.101.199.29:54310/user/hadoop-node1/flume-test/ (http://107.101.199.29:54310/user/hadoop-node1/flume-test/)
> agent1.sinks.HDFS.hdfs.file.Type = DataStream
>
>
> # Finally, now that we've defined all of our components, tell
> # agent1 which ones we want to activate.
> agent1.channels = ch1
> agent1.sources = avro-source1
> agent1.sinks = HDFS
>
> 2) it is a single node hadoop setup. command executed for flume are
> bin/flume-ng agent --conf ./conf/ -f conf/flumeAVRO_HDFS.conf -n agent1 (in one console)
> bin/flume-ng avro-client --conf conf -H 107.108.199.29 -p 41414 -F /home/hadoop-node1/Desktop/my.txt (in other console for Avro CLient) [ also tried command with -H localhost option), but same error.
>
> 3) hadoop version 0.20 is used
>
>
>
>
> With Regards,
> Amit
>
>
> Attachments:
> - error.log
>
FlumeNG Error while going with Avro (Source) and HDFS (Sink)
Posted by Amit Handa <am...@gmail.com>.
Hi,
1) While going for Avro (as Source) , and HDFS as Sink. I am getting error
"[ERROR -
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:125)]
RPC connection error :". Kindly help in resolving it
1 a) flume.conf file is as follows
# Define a memory channel called ch1 on agent1
agent1.channels.ch1.type = memory
# Define an Avro source called avro-source1 on agent1 and tell it
# to bind to 0.0.0.0:41414. Connect it to channel ch1.
agent1.sources.avro-source1.channels = ch1
agent1.sources.avro-source1.type = avro
agent1.sources.avro-source1.bind = 107.101.199.29
agent1.sources.avro-source1.port = 41414
# Define a hdfs sink that simply logs all events it receives
# and connect it to the other end of the same channel.
agent1.sinks.HDFS.channel = ch1
agent1.sinks.HDFS.type = hdfs
agent1.sinks.HDFS.hdfs.path = hdfs://
107.101.199.29:54310/user/hadoop-node1/flume-test/
agent1.sinks.HDFS.hdfs.file.Type = DataStream
# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = ch1
agent1.sources = avro-source1
agent1.sinks = HDFS
2) it is a single node hadoop setup. command executed for flume are
bin/flume-ng agent --conf ./conf/ -f conf/flumeAVRO_HDFS.conf
-n agent1 (in one console)
bin/flume-ng avro-client --conf conf -H 107.108.199.29 -p 41414
-F /home/hadoop-node1/Desktop/my.txt (in other console for Avro CLient) [
also tried command with -H localhost option), but same error.
3) hadoop version 0.20 is used
With Regards,
Amit