You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sai Sai <sa...@yahoo.in> on 2013/04/13 03:45:28 UTC
Re: 100K Maps scenario
Just a follow up to see if anyone can shed some light on this:
My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
Will appreciate any thoughts on this.
Thanks
Sai
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
Sent: Friday, 12 April 2013 1:37 PM
Subject: Re: Does a Map task run 3 times on 3 TTs or just once
Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
Plesae pour some light.
Thanks
Sai
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org