You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Siva Gudavalli <si...@synechron.com> on 2012/11/01 16:13:41 UTC
avro client fails to connect to flume
hi,
i am tryint to use avro client to source my log file to avro-source.
flume-source -> avro source
flume-channel -> memory
flume-sink -> hdfs
all of the services are up; but when i am calling avro client its failing connection to flume.
please help!
environment:
hadoop 0.20.2
flume 1.2
logs from agent:
2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source avro-AppSrv-source started.
2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370 => /127.0.0.1:41414] OPEN
2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:43370
2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370 :> /127.0.0.1:41414] DISCONNECTED
2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370 :> /127.0.0.1:41414] UNBOUND
2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370 :> /127.0.0.1:41414] CLOSED
logs from avro client:
Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
+ exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
Re: avro client fails to connect to flume
Posted by Alexander Lorenz <wg...@gmail.com>.
Hi,
> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
> Caused by: java.net.ConnectException: Connection refused
Can you telnet to this port? Do you have a firewall running? And did you try to bound to eth0 or 0.0.0.0?
cheers
- Alex
On Nov 2, 2012, at 5:16 AM, Siva Gudavalli <si...@synechron.com> wrote:
> Hi Hari,
>
> Agent is up and the issue is with avro client that comes along with flume...
> it is not able to establish connection with flume.
>
> these are the logs for my avro client
>
>> Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
>> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
>> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
>> 2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
>> 2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
>> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
>> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
>> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
>> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
>> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
>> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
>> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>> ... 5 more
>> Caused by: java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
>> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
>> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
>> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> 2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>>
>
>
> ________________________________________
> From: Hari Shreedharan [hshreedharan@cloudera.com]
> Sent: Friday, November 02, 2012 8:18 AM
> To: user@flume.apache.org
> Subject: Re: avro client fails to connect to flume
>
> Please send your logs also. It will help us understand if there is an
> error during startup.
>
> Hari
>
> On Thu, Nov 1, 2012 at 6:44 PM, Siva Gudavalli
> <si...@synechron.com> wrote:
>> hi Roshan,
>>
>> here is the config....
>>
>> agent.sources = avro-AppSrv-source
>> agent.sinks = hdfs-Cluster-sink
>> agent.channels = mem-channel
>>
>> # set channel for sources, sinks
>> agent.sources.avro-AppSrv-source.channels = mem-channel
>> agent.sources.hdfs-Cluster-sink.channel = mem-channel
>>
>> # set source properties
>> agent.sources.avro-AppSrv-source.type = avro
>> agent.sources.avro-AppSrv-source.bind = localhost
>> agent.sources.avro-AppSrv-source.port = 41414
>>
>> # set channel properties
>> agent.channels.mem-channel.type = memory
>> agent.channels.mem-channel.capacity = 10000
>> agent.channels.mem-channel.transactionCapacity = 10000
>>
>> # set sink properties
>> agent.sinks.hdfs-Cluster-sink.type = hdfs
>> agent.sinks.hdfs-Cluster-sink.hdfs.path = hdfs://localhost:54310/flume
>>
>> ________________________________________
>> From: Roshan Naik [roshan@hortonworks.com]
>> Sent: Friday, November 02, 2012 4:09 AM
>> To: user@flume.apache.org
>> Subject: Re: avro client fails to connect to flume
>>
>> Could you provide your flume configs ?
>> -roshan
>>
>> On Thu, Nov 1, 2012 at 8:13 AM, Siva Gudavalli <si...@synechron.com>> wrote:
>> hi,
>>
>> i am tryint to use avro client to source my log file to avro-source.
>>
>> flume-source -> avro source
>> flume-channel -> memory
>> flume-sink -> hdfs
>>
>> all of the services are up; but when i am calling avro client its failing connection to flume.
>>
>> please help!
>>
>> environment:
>>
>> hadoop 0.20.2
>> flume 1.2
>>
>> logs from agent:
>>
>> 2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source avro-AppSrv-source started.
>> 2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] OPEN
>> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] BOUND: /127.0.0.1:41414<http://127.0.0.1:41414>
>> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] CONNECTED: /127.0.0.1:43370<http://127.0.0.1:43370>
>> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] DISCONNECTED
>> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] UNBOUND
>> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] CLOSED
>>
>> logs from avro client:
>>
>> Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
>> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
>> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
>> 2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
>> 2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
>> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
>> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
>> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
>> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
>> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
>> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
>> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>> ... 5 more
>> Caused by: java.net.ConnectException: Connection refused
>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
>> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
>> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
>> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> at java.lang.Thread.run(Thread.java:722)
>> 2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>>
--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF
RE: avro client fails to connect to flume
Posted by Siva Gudavalli <si...@synechron.com>.
Hi Hari,
Agent is up and the issue is with avro client that comes along with flume...
it is not able to establish connection with flume.
these are the logs for my avro client
> Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
> 2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
> 2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>
________________________________________
From: Hari Shreedharan [hshreedharan@cloudera.com]
Sent: Friday, November 02, 2012 8:18 AM
To: user@flume.apache.org
Subject: Re: avro client fails to connect to flume
Please send your logs also. It will help us understand if there is an
error during startup.
Hari
On Thu, Nov 1, 2012 at 6:44 PM, Siva Gudavalli
<si...@synechron.com> wrote:
> hi Roshan,
>
> here is the config....
>
> agent.sources = avro-AppSrv-source
> agent.sinks = hdfs-Cluster-sink
> agent.channels = mem-channel
>
> # set channel for sources, sinks
> agent.sources.avro-AppSrv-source.channels = mem-channel
> agent.sources.hdfs-Cluster-sink.channel = mem-channel
>
> # set source properties
> agent.sources.avro-AppSrv-source.type = avro
> agent.sources.avro-AppSrv-source.bind = localhost
> agent.sources.avro-AppSrv-source.port = 41414
>
> # set channel properties
> agent.channels.mem-channel.type = memory
> agent.channels.mem-channel.capacity = 10000
> agent.channels.mem-channel.transactionCapacity = 10000
>
> # set sink properties
> agent.sinks.hdfs-Cluster-sink.type = hdfs
> agent.sinks.hdfs-Cluster-sink.hdfs.path = hdfs://localhost:54310/flume
>
> ________________________________________
> From: Roshan Naik [roshan@hortonworks.com]
> Sent: Friday, November 02, 2012 4:09 AM
> To: user@flume.apache.org
> Subject: Re: avro client fails to connect to flume
>
> Could you provide your flume configs ?
> -roshan
>
> On Thu, Nov 1, 2012 at 8:13 AM, Siva Gudavalli <si...@synechron.com>> wrote:
> hi,
>
> i am tryint to use avro client to source my log file to avro-source.
>
> flume-source -> avro source
> flume-channel -> memory
> flume-sink -> hdfs
>
> all of the services are up; but when i am calling avro client its failing connection to flume.
>
> please help!
>
> environment:
>
> hadoop 0.20.2
> flume 1.2
>
> logs from agent:
>
> 2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source avro-AppSrv-source started.
> 2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] OPEN
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] BOUND: /127.0.0.1:41414<http://127.0.0.1:41414>
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] CONNECTED: /127.0.0.1:43370<http://127.0.0.1:43370>
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] DISCONNECTED
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] UNBOUND
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] CLOSED
>
> logs from avro client:
>
> Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
> 2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
> 2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>
Re: avro client fails to connect to flume
Posted by Hari Shreedharan <hs...@cloudera.com>.
Please send your logs also. It will help us understand if there is an
error during startup.
Hari
On Thu, Nov 1, 2012 at 6:44 PM, Siva Gudavalli
<si...@synechron.com> wrote:
> hi Roshan,
>
> here is the config....
>
> agent.sources = avro-AppSrv-source
> agent.sinks = hdfs-Cluster-sink
> agent.channels = mem-channel
>
> # set channel for sources, sinks
> agent.sources.avro-AppSrv-source.channels = mem-channel
> agent.sources.hdfs-Cluster-sink.channel = mem-channel
>
> # set source properties
> agent.sources.avro-AppSrv-source.type = avro
> agent.sources.avro-AppSrv-source.bind = localhost
> agent.sources.avro-AppSrv-source.port = 41414
>
> # set channel properties
> agent.channels.mem-channel.type = memory
> agent.channels.mem-channel.capacity = 10000
> agent.channels.mem-channel.transactionCapacity = 10000
>
> # set sink properties
> agent.sinks.hdfs-Cluster-sink.type = hdfs
> agent.sinks.hdfs-Cluster-sink.hdfs.path = hdfs://localhost:54310/flume
>
> ________________________________________
> From: Roshan Naik [roshan@hortonworks.com]
> Sent: Friday, November 02, 2012 4:09 AM
> To: user@flume.apache.org
> Subject: Re: avro client fails to connect to flume
>
> Could you provide your flume configs ?
> -roshan
>
> On Thu, Nov 1, 2012 at 8:13 AM, Siva Gudavalli <si...@synechron.com>> wrote:
> hi,
>
> i am tryint to use avro client to source my log file to avro-source.
>
> flume-source -> avro source
> flume-channel -> memory
> flume-sink -> hdfs
>
> all of the services are up; but when i am calling avro client its failing connection to flume.
>
> please help!
>
> environment:
>
> hadoop 0.20.2
> flume 1.2
>
> logs from agent:
>
> 2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source avro-AppSrv-source started.
> 2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] OPEN
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] BOUND: /127.0.0.1:41414<http://127.0.0.1:41414>
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] CONNECTED: /127.0.0.1:43370<http://127.0.0.1:43370>
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] DISCONNECTED
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] UNBOUND
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] CLOSED
>
> logs from avro client:
>
> Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
> 2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
> 2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
>
RE: avro client fails to connect to flume
Posted by Siva Gudavalli <si...@synechron.com>.
hi Roshan,
here is the config....
agent.sources = avro-AppSrv-source
agent.sinks = hdfs-Cluster-sink
agent.channels = mem-channel
# set channel for sources, sinks
agent.sources.avro-AppSrv-source.channels = mem-channel
agent.sources.hdfs-Cluster-sink.channel = mem-channel
# set source properties
agent.sources.avro-AppSrv-source.type = avro
agent.sources.avro-AppSrv-source.bind = localhost
agent.sources.avro-AppSrv-source.port = 41414
# set channel properties
agent.channels.mem-channel.type = memory
agent.channels.mem-channel.capacity = 10000
agent.channels.mem-channel.transactionCapacity = 10000
# set sink properties
agent.sinks.hdfs-Cluster-sink.type = hdfs
agent.sinks.hdfs-Cluster-sink.hdfs.path = hdfs://localhost:54310/flume
________________________________________
From: Roshan Naik [roshan@hortonworks.com]
Sent: Friday, November 02, 2012 4:09 AM
To: user@flume.apache.org
Subject: Re: avro client fails to connect to flume
Could you provide your flume configs ?
-roshan
On Thu, Nov 1, 2012 at 8:13 AM, Siva Gudavalli <si...@synechron.com>> wrote:
hi,
i am tryint to use avro client to source my log file to avro-source.
flume-source -> avro source
flume-channel -> memory
flume-sink -> hdfs
all of the services are up; but when i am calling avro client its failing connection to flume.
please help!
environment:
hadoop 0.20.2
flume 1.2
logs from agent:
2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source avro-AppSrv-source started.
2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] OPEN
2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] BOUND: /127.0.0.1:41414<http://127.0.0.1:41414>
2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> => /127.0.0.1:41414<http://127.0.0.1:41414>] CONNECTED: /127.0.0.1:43370<http://127.0.0.1:43370>
2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] DISCONNECTED
2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] UNBOUND
2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /127.0.0.1:43370<http://127.0.0.1:43370> :> /127.0.0.1:41414<http://127.0.0.1:41414>] CLOSED
logs from avro client:
Info: Sourcing environment configuration script /usr/local/flume/conf/flume-env.sh
Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
+ exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf' -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
2012-11-01 20:25:50,176 (main) [DEBUG - org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)] Batch size string = 5
2012-11-01 20:25:50,284 (main) [ERROR - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)] Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:41414<http://127.0.0.1:41414>
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2012-11-01 20:25:50,288 (main) [DEBUG - org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)] Exiting
Re: avro client fails to connect to flume
Posted by Roshan Naik <ro...@hortonworks.com>.
Could you provide your flume configs ?
-roshan
On Thu, Nov 1, 2012 at 8:13 AM, Siva Gudavalli <siva.gudavalli@synechron.com
> wrote:
> hi,
>
> i am tryint to use avro client to source my log file to avro-source.
>
> flume-source -> avro source
> flume-channel -> memory
> flume-sink -> hdfs
>
> all of the services are up; but when i am calling avro client its failing
> connection to flume.
>
> please help!
>
> environment:
>
> hadoop 0.20.2
> flume 1.2
>
> logs from agent:
>
> 2012-11-01 20:03:49,977 INFO source.AvroSource: Avro source
> avro-AppSrv-source started.
> 2012-11-01 20:04:01,685 INFO ipc.NettyServer: [id: 0x00e71be2, /
> 127.0.0.1:43370 => /127.0.0.1:41414] OPEN
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /
> 127.0.0.1:43370 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
> 2012-11-01 20:04:01,689 INFO ipc.NettyServer: [id: 0x00e71be2, /
> 127.0.0.1:43370 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:43370
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /
> 127.0.0.1:43370 :> /127.0.0.1:41414] DISCONNECTED
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /
> 127.0.0.1:43370 :> /127.0.0.1:41414] UNBOUND
> 2012-11-01 20:04:02,562 INFO ipc.NettyServer: [id: 0x00e71be2, /
> 127.0.0.1:43370 :> /127.0.0.1:41414] CLOSED
>
> logs from avro client:
>
> Info: Sourcing environment configuration script
> /usr/local/flume/conf/flume-env.sh
> Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop)
> for HDFS access
> + exec /usr/lib/jvm/java-7-openjdk-i386/bin/java -Xms1024m -Xmx2048m
> -Dcom.sun.management.jmxremote -Dflume.root.logger=DEBUG,console -cp
> '/usr/local/flume/conf:/usr/local/flume/lib/*:/usr/local/flume/conf'
> -Djava.library.path=:/usr/local/hadoop/bin/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H localhost -p 41414 -F
> /usr/local/hbase/logs/hbase-hduser-master-172.20.204.119.log
> 2012-11-01 20:25:50,176 (main) [DEBUG -
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
> Batch size string = 5
> 2012-11-01 20:25:50,284 (main) [ERROR -
> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:76)]
> Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
> port: 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at
> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:168)
> at
> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:68)
> Caused by: java.io.IOException: Error connecting to localhost/
> 127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198)
> at
> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2012-11-01 20:25:50,288 (main) [DEBUG -
> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:81)]
> Exiting
>