You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Sai Sai <sa...@yahoo.in> on 2013/04/12 10:07:40 UTC
Re: Does a Map task run 3 times on 3 TTs or just once
Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
Plesae pour some light.
Thanks
Sai
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a
separate machine
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS - hadoop distributed file system
as it stands a file system .. first basic question you will need to search
is do you need a process to run a file system?
when you find an answer to that second question will be
will a single process be enough for a distributed system ? meaning sub
components of the system may exist on different machines
namenode and datanode combined make hdfs. combining all of their processes
you make hdfs.
namenode is master for the hdfs which keeps the file system image in memory
when it starts it loads it up in memory and serves all requests from memory
there on. There are steps taken to save the FSImage to disk. You can read
about it in detail in hdfs architecture.
when you put a file in hdfs .. it may or may not go to a single machine.
Namenode never stores the data files. it just stores the metadata for the
hdfs.
so when you load a file it will be going to datanode and the file
information will be going to namenode. depending on the size it will be
split in multiple blocks and then multiple blocks may land on multiple
datanodes. If your filesize is less than or exactly equal to block size you
can find out which datanode it is located. else there is no guarantee that
file will be only on single node only if you have fully distributed mode
PS: this is my understanding. Others may correct me as well
On Fri, Apr 12, 2013 at 2:00 PM, Sai Sai <sa...@yahoo.in> wrote:
> A few basic questions:
>
> Will HDFS refer to the memory of NameNode & DataNode or is it a separate
> machine.
>
> For NameNode, DataNode and others there is a process associated with each
> of em.
> But no process is for HDFS, wondering why? I understand that fsImage has
> the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT
> needs to get file info will they just look into the fsImage.
>
> When we put a file in HDFS is it possible to look/find in which node
> (NN/DN) it physically sits.
>
> Any help is appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a
separate machine
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS - hadoop distributed file system
as it stands a file system .. first basic question you will need to search
is do you need a process to run a file system?
when you find an answer to that second question will be
will a single process be enough for a distributed system ? meaning sub
components of the system may exist on different machines
namenode and datanode combined make hdfs. combining all of their processes
you make hdfs.
namenode is master for the hdfs which keeps the file system image in memory
when it starts it loads it up in memory and serves all requests from memory
there on. There are steps taken to save the FSImage to disk. You can read
about it in detail in hdfs architecture.
when you put a file in hdfs .. it may or may not go to a single machine.
Namenode never stores the data files. it just stores the metadata for the
hdfs.
so when you load a file it will be going to datanode and the file
information will be going to namenode. depending on the size it will be
split in multiple blocks and then multiple blocks may land on multiple
datanodes. If your filesize is less than or exactly equal to block size you
can find out which datanode it is located. else there is no guarantee that
file will be only on single node only if you have fully distributed mode
PS: this is my understanding. Others may correct me as well
On Fri, Apr 12, 2013 at 2:00 PM, Sai Sai <sa...@yahoo.in> wrote:
> A few basic questions:
>
> Will HDFS refer to the memory of NameNode & DataNode or is it a separate
> machine.
>
> For NameNode, DataNode and others there is a process associated with each
> of em.
> But no process is for HDFS, wondering why? I understand that fsImage has
> the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT
> needs to get file info will they just look into the fsImage.
>
> When we put a file in HDFS is it possible to look/find in which node
> (NN/DN) it physically sits.
>
> Any help is appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a
separate machine
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS - hadoop distributed file system
as it stands a file system .. first basic question you will need to search
is do you need a process to run a file system?
when you find an answer to that second question will be
will a single process be enough for a distributed system ? meaning sub
components of the system may exist on different machines
namenode and datanode combined make hdfs. combining all of their processes
you make hdfs.
namenode is master for the hdfs which keeps the file system image in memory
when it starts it loads it up in memory and serves all requests from memory
there on. There are steps taken to save the FSImage to disk. You can read
about it in detail in hdfs architecture.
when you put a file in hdfs .. it may or may not go to a single machine.
Namenode never stores the data files. it just stores the metadata for the
hdfs.
so when you load a file it will be going to datanode and the file
information will be going to namenode. depending on the size it will be
split in multiple blocks and then multiple blocks may land on multiple
datanodes. If your filesize is less than or exactly equal to block size you
can find out which datanode it is located. else there is no guarantee that
file will be only on single node only if you have fully distributed mode
PS: this is my understanding. Others may correct me as well
On Fri, Apr 12, 2013 at 2:00 PM, Sai Sai <sa...@yahoo.in> wrote:
> A few basic questions:
>
> Will HDFS refer to the memory of NameNode & DataNode or is it a separate
> machine.
>
> For NameNode, DataNode and others there is a process associated with each
> of em.
> But no process is for HDFS, wondering why? I understand that fsImage has
> the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT
> needs to get file info will they just look into the fsImage.
>
> When we put a file in HDFS is it possible to look/find in which node
> (NN/DN) it physically sits.
>
> Any help is appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a
separate machine
Posted by Nitin Pawar <ni...@gmail.com>.
HDFS - hadoop distributed file system
as it stands a file system .. first basic question you will need to search
is do you need a process to run a file system?
when you find an answer to that second question will be
will a single process be enough for a distributed system ? meaning sub
components of the system may exist on different machines
namenode and datanode combined make hdfs. combining all of their processes
you make hdfs.
namenode is master for the hdfs which keeps the file system image in memory
when it starts it loads it up in memory and serves all requests from memory
there on. There are steps taken to save the FSImage to disk. You can read
about it in detail in hdfs architecture.
when you put a file in hdfs .. it may or may not go to a single machine.
Namenode never stores the data files. it just stores the metadata for the
hdfs.
so when you load a file it will be going to datanode and the file
information will be going to namenode. depending on the size it will be
split in multiple blocks and then multiple blocks may land on multiple
datanodes. If your filesize is less than or exactly equal to block size you
can find out which datanode it is located. else there is no guarantee that
file will be only on single node only if you have fully distributed mode
PS: this is my understanding. Others may correct me as well
On Fri, Apr 12, 2013 at 2:00 PM, Sai Sai <sa...@yahoo.in> wrote:
> A few basic questions:
>
> Will HDFS refer to the memory of NameNode & DataNode or is it a separate
> machine.
>
> For NameNode, DataNode and others there is a process associated with each
> of em.
> But no process is for HDFS, wondering why? I understand that fsImage has
> the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT
> needs to get file info will they just look into the fsImage.
>
> When we put a file in HDFS is it possible to look/find in which node
> (NN/DN) it physically sits.
>
> Any help is appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine
Posted by Sai Sai <sa...@yahoo.in>.
A few basic questions:
Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine.
For NameNode, DataNode and others there is a process associated with each of em.
But no process is for HDFS, wondering why? I understand that fsImage has the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT needs to get file info will they just look into the fsImage.
When we put a file in HDFS is it possible to look/find in which node (NN/DN) it physically sits.
Any help is appreciated.
Thanks
Sai
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine
Posted by Sai Sai <sa...@yahoo.in>.
A few basic questions:
Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine.
For NameNode, DataNode and others there is a process associated with each of em.
But no process is for HDFS, wondering why? I understand that fsImage has the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT needs to get file info will they just look into the fsImage.
When we put a file in HDFS is it possible to look/find in which node (NN/DN) it physically sits.
Any help is appreciated.
Thanks
Sai
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine
Posted by Sai Sai <sa...@yahoo.in>.
A few basic questions:
Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine.
For NameNode, DataNode and others there is a process associated with each of em.
But no process is for HDFS, wondering why? I understand that fsImage has the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT needs to get file info will they just look into the fsImage.
When we put a file in HDFS is it possible to look/find in which node (NN/DN) it physically sits.
Any help is appreciated.
Thanks
Sai
Re: How to find the num of Mappers
Posted by Nitin Pawar <ni...@gmail.com>.
your question is answered here
http://wiki.apache.org/hadoop/HowManyMapsAndReduces
To answer first part of your question,
it is not mandatory to run all the maps of a given job at a single time.
Maps are executed as and when the map slots are available on the
tasktrackers
On Fri, Apr 12, 2013 at 1:51 PM, Sai Sai <sa...@yahoo.in> wrote:
> If we have a 640 MB data file and have 3 Data Nodes in a cluster.
> The file can be split into 10 Blocks and starts the Mappers M1, M2, M3
> first.
> As each one completes the task M4 and so on will be run.
> It appears like it is not necessary to run all the 10 Map tasks in
> parallel at once.
> Just wondering if this is right assumption.
> What if we have 10 TB of data file with 3 Data Nodes, how to find the
> number of mappers that will be created.
> Thanks
> Sai
>
--
Nitin Pawar
Re: How to find the num of Mappers
Posted by Nitin Pawar <ni...@gmail.com>.
your question is answered here
http://wiki.apache.org/hadoop/HowManyMapsAndReduces
To answer first part of your question,
it is not mandatory to run all the maps of a given job at a single time.
Maps are executed as and when the map slots are available on the
tasktrackers
On Fri, Apr 12, 2013 at 1:51 PM, Sai Sai <sa...@yahoo.in> wrote:
> If we have a 640 MB data file and have 3 Data Nodes in a cluster.
> The file can be split into 10 Blocks and starts the Mappers M1, M2, M3
> first.
> As each one completes the task M4 and so on will be run.
> It appears like it is not necessary to run all the 10 Map tasks in
> parallel at once.
> Just wondering if this is right assumption.
> What if we have 10 TB of data file with 3 Data Nodes, how to find the
> number of mappers that will be created.
> Thanks
> Sai
>
--
Nitin Pawar
Re: How to find the num of Mappers
Posted by Nitin Pawar <ni...@gmail.com>.
your question is answered here
http://wiki.apache.org/hadoop/HowManyMapsAndReduces
To answer first part of your question,
it is not mandatory to run all the maps of a given job at a single time.
Maps are executed as and when the map slots are available on the
tasktrackers
On Fri, Apr 12, 2013 at 1:51 PM, Sai Sai <sa...@yahoo.in> wrote:
> If we have a 640 MB data file and have 3 Data Nodes in a cluster.
> The file can be split into 10 Blocks and starts the Mappers M1, M2, M3
> first.
> As each one completes the task M4 and so on will be run.
> It appears like it is not necessary to run all the 10 Map tasks in
> parallel at once.
> Just wondering if this is right assumption.
> What if we have 10 TB of data file with 3 Data Nodes, how to find the
> number of mappers that will be created.
> Thanks
> Sai
>
--
Nitin Pawar
Re: Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine
Posted by Sai Sai <sa...@yahoo.in>.
A few basic questions:
Will HDFS refer to the memory of NameNode & DataNode or is it a separate machine.
For NameNode, DataNode and others there is a process associated with each of em.
But no process is for HDFS, wondering why? I understand that fsImage has the meta data of the HDFS, so when NameNode or DataNode or JobTracker/TT needs to get file info will they just look into the fsImage.
When we put a file in HDFS is it possible to look/find in which node (NN/DN) it physically sits.
Any help is appreciated.
Thanks
Sai
Re: How to find the num of Mappers
Posted by Nitin Pawar <ni...@gmail.com>.
your question is answered here
http://wiki.apache.org/hadoop/HowManyMapsAndReduces
To answer first part of your question,
it is not mandatory to run all the maps of a given job at a single time.
Maps are executed as and when the map slots are available on the
tasktrackers
On Fri, Apr 12, 2013 at 1:51 PM, Sai Sai <sa...@yahoo.in> wrote:
> If we have a 640 MB data file and have 3 Data Nodes in a cluster.
> The file can be split into 10 Blocks and starts the Mappers M1, M2, M3
> first.
> As each one completes the task M4 and so on will be run.
> It appears like it is not necessary to run all the 10 Map tasks in
> parallel at once.
> Just wondering if this is right assumption.
> What if we have 10 TB of data file with 3 Data Nodes, how to find the
> number of mappers that will be created.
> Thanks
> Sai
>
--
Nitin Pawar
Re: How to find the num of Mappers
Posted by Sai Sai <sa...@yahoo.in>.
If we have a 640 MB data file and have 3 Data Nodes in a cluster.
The file can be split into 10 Blocks and starts the Mappers M1, M2, M3 first.
As each one completes the task M4 and so on will be run.
It appears like it is not necessary to run all the 10 Map tasks in parallel at once.
Just wondering if this is right assumption.
What if we have 10 TB of data file with 3 Data Nodes, how to find the number of mappers that will be created.
Thanks
Sai
Re: 10 TB of a data file.
Posted by Nitin Pawar <ni...@gmail.com>.
>From wikipedia
The actual amount of disk
space<http://en.wikipedia.org/wiki/Computer_data_storage>consumed by
the file depends on the file
system <http://en.wikipedia.org/wiki/File_system>. The maximum file size a
file system supports depends on the number of
bits<http://en.wikipedia.org/wiki/Bit>reserved to store size
information and the total size of the file system
more of it you can read on http://en.wikipedia.org/wiki/File_size
On Fri, Apr 12, 2013 at 1:40 PM, Sai Sai <sa...@yahoo.in> wrote:
> In real world can a file be of this big size as 10 TB?
> Will the data be put into a txt file or what kind of a file?
> If someone would like to open such a big file to look at the content will
> OS support opening such big files?
> If not how to handle this kind of scenario?
> Any input will be appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: 10 TB of a data file.
Posted by Nitin Pawar <ni...@gmail.com>.
>From wikipedia
The actual amount of disk
space<http://en.wikipedia.org/wiki/Computer_data_storage>consumed by
the file depends on the file
system <http://en.wikipedia.org/wiki/File_system>. The maximum file size a
file system supports depends on the number of
bits<http://en.wikipedia.org/wiki/Bit>reserved to store size
information and the total size of the file system
more of it you can read on http://en.wikipedia.org/wiki/File_size
On Fri, Apr 12, 2013 at 1:40 PM, Sai Sai <sa...@yahoo.in> wrote:
> In real world can a file be of this big size as 10 TB?
> Will the data be put into a txt file or what kind of a file?
> If someone would like to open such a big file to look at the content will
> OS support opening such big files?
> If not how to handle this kind of scenario?
> Any input will be appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: 10 TB of a data file.
Posted by Nitin Pawar <ni...@gmail.com>.
>From wikipedia
The actual amount of disk
space<http://en.wikipedia.org/wiki/Computer_data_storage>consumed by
the file depends on the file
system <http://en.wikipedia.org/wiki/File_system>. The maximum file size a
file system supports depends on the number of
bits<http://en.wikipedia.org/wiki/Bit>reserved to store size
information and the total size of the file system
more of it you can read on http://en.wikipedia.org/wiki/File_size
On Fri, Apr 12, 2013 at 1:40 PM, Sai Sai <sa...@yahoo.in> wrote:
> In real world can a file be of this big size as 10 TB?
> Will the data be put into a txt file or what kind of a file?
> If someone would like to open such a big file to look at the content will
> OS support opening such big files?
> If not how to handle this kind of scenario?
> Any input will be appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: How to find the num of Mappers
Posted by Sai Sai <sa...@yahoo.in>.
If we have a 640 MB data file and have 3 Data Nodes in a cluster.
The file can be split into 10 Blocks and starts the Mappers M1, M2, M3 first.
As each one completes the task M4 and so on will be run.
It appears like it is not necessary to run all the 10 Map tasks in parallel at once.
Just wondering if this is right assumption.
What if we have 10 TB of data file with 3 Data Nodes, how to find the number of mappers that will be created.
Thanks
Sai
Re: How to find the num of Mappers
Posted by Sai Sai <sa...@yahoo.in>.
If we have a 640 MB data file and have 3 Data Nodes in a cluster.
The file can be split into 10 Blocks and starts the Mappers M1, M2, M3 first.
As each one completes the task M4 and so on will be run.
It appears like it is not necessary to run all the 10 Map tasks in parallel at once.
Just wondering if this is right assumption.
What if we have 10 TB of data file with 3 Data Nodes, how to find the number of mappers that will be created.
Thanks
Sai
Re: How to find the num of Mappers
Posted by Sai Sai <sa...@yahoo.in>.
If we have a 640 MB data file and have 3 Data Nodes in a cluster.
The file can be split into 10 Blocks and starts the Mappers M1, M2, M3 first.
As each one completes the task M4 and so on will be run.
It appears like it is not necessary to run all the 10 Map tasks in parallel at once.
Just wondering if this is right assumption.
What if we have 10 TB of data file with 3 Data Nodes, how to find the number of mappers that will be created.
Thanks
Sai
Re: 10 TB of a data file.
Posted by Nitin Pawar <ni...@gmail.com>.
>From wikipedia
The actual amount of disk
space<http://en.wikipedia.org/wiki/Computer_data_storage>consumed by
the file depends on the file
system <http://en.wikipedia.org/wiki/File_system>. The maximum file size a
file system supports depends on the number of
bits<http://en.wikipedia.org/wiki/Bit>reserved to store size
information and the total size of the file system
more of it you can read on http://en.wikipedia.org/wiki/File_size
On Fri, Apr 12, 2013 at 1:40 PM, Sai Sai <sa...@yahoo.in> wrote:
> In real world can a file be of this big size as 10 TB?
> Will the data be put into a txt file or what kind of a file?
> If someone would like to open such a big file to look at the content will
> OS support opening such big files?
> If not how to handle this kind of scenario?
> Any input will be appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: 10 TB of a data file.
Posted by Sai Sai <sa...@yahoo.in>.
In real world can a file be of this big size as 10 TB?
Will the data be put into a txt file or what kind of a file?
If someone would like to open such a big file to look at the content will OS support opening such big files?
If not how to handle this kind of scenario?
Any input will be appreciated.
Thanks
Sai
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
Reattempting to post this to the flume ML after subscribing...
On 21 May 2013 16:29, Paul Wilkinson <pa...@gmail.com> wrote:
> This could be a myriad of things, but some points to note:
>
> 1. Flume has its own mailing list - I've moved your post there instead.
> 2. Your command line suggests connecting to localhost:4000, but the log
> shows ubuntu:41414 instead (different attempts maybe?)
> 3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
>
> I'd suggest taking a look at the logs for the agent itself, rather than
> the Avro Client - chances are the agent isn't starting up due to the config
> problems.
>
> Paul
>
>
> On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
>
> Just a friendly follow up to see if anyone has any suggestions for the
> issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
>
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Paul Wilkinson <pa...@gmail.com>.
This could be a myriad of things, but some points to note:
1. Flume has its own mailing list - I've moved your post there instead.
2. Your command line suggests connecting to localhost:4000, but the log shows ubuntu:41414 instead (different attempts maybe?)
3. The correct type for file roll sink is "file_roll", not "FILE_ROLL".
I'd suggest taking a look at the logs for the agent itself, rather than the Avro Client - chances are the agent isn't starting up due to the config problems.
Paul
On 21 May 2013, at 08:47, Sai Sai <sa...@yahoo.in> wrote:
> Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
> Any help is appreciated.
> Thanks
> Sai
>
> On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Just a friendly follow up to see if anyone has any suggestions for the issue with port given below.
Any help is appreciated.
Thanks
Sai
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Lenin
Thanks for your reply.
Here is the 1st sample which works, i am not sure if you r referring to this:
-------------------------------
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
-------------------------------
Please let me know if u have any suggestions.
Thanks
Sai
________________________________
From: Lenin Raj <em...@gmail.com>
To: user@hadoop.apache.org
Sent: Monday, 20 May 2013 5:54 PM
Subject: Re: Flume port issue
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Not sure if this is the right group to ask questions about flume:
>
>
>I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
>---------------------------
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>---------------------------
>
>
>
>Here r the steps i have followed:
>
>
>Step 1: Here is my agent3.conf created in the flume/conf dir:
>
>
>******
>agent3.sources = avrosource
>agent3.sinks = filesink
>agent3.channels = jdbcchannel
>
>
>agent3.sources.avrosource.type = avro
>agent3.sources.avrosource.bind = localhost
>agent3.sources.avrosource.port = 4000
>agent3.sources.avrosource.threads = 5
>
>
>agent3.sinks.filesink.type = FILE_ROLL
>agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
>agent3.sinks.filesink.sink.rollInterval = 0
>
>
>agent3.channels.jdbcchannel.type = jdbc
>
>
>agent3.sources.avrosource.channels = jdbcchannel
>agent3.sinks.filesink.channel = jdbcchannel
>
>
>******
>
>
>
>Step 2: Then i have saved it successfully and created a new test file like this:
>
>
>Step 3: echo "Hello World" > /home/satish/message3
>
>
>Step 4: Tried executing this command:
>
>
>./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
>
>I get this exception below, please help:
>
>
>------------------------------
>
>
>Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
>13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
>org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
>at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>... 5 more
>Caused by: java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
>
>Please help.
>Thanks
>Sai
>
>
>
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Lenin Raj <em...@gmail.com>.
Sai, Are you able to run the netcat flume sample?
--
Lenin.
On May 20, 2013 5:40 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Not sure if this is the right group to ask questions about flume:
>
> I am getting an exception about unable to open a port in flume when trying
> to create a remote agent, more details below:
> ---------------------------
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> ---------------------------
>
> Here r the steps i have followed:
>
> Step 1: Here is my agent3.conf created in the flume/conf dir:
>
> ******
> agent3.sources = avrosource
> agent3.sinks = filesink
> agent3.channels = jdbcchannel
>
> agent3.sources.avrosource.type = avro
> agent3.sources.avrosource.bind = localhost
> agent3.sources.avrosource.port = 4000
> agent3.sources.avrosource.threads = 5
>
> agent3.sinks.filesink.type = FILE_ROLL
> agent3.sinks.filesink.sink.directory =
> /home/satish/work/apache-flume-1.3.1-bin/files
> agent3.sinks.filesink.sink.rollInterval = 0
>
> agent3.channels.jdbcchannel.type = jdbc
>
> agent3.sources.avrosource.channels = jdbcchannel
> agent3.sinks.filesink.channel = jdbcchannel
>
> ******
>
> Step 2: Then i have saved it successfully and created a new test file like
> this:
>
> Step 3: echo "Hello World" > /home/satish/message3
>
> Step 4: Tried executing this command:
>
> ./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
>
> I get this exception below, please help:
>
> ------------------------------
>
> Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32
> org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F
> /usr/logs/log.10
> 13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to
> Flume. Exception follows.
> org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port:
> 41414 }: RPC connection error
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
> at
> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
> at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
> at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
> Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
> ... 5 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
>
> Please help.
> Thanks
> Sai
>
>
>
>
>
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: Flume port issue
Posted by Sai Sai <sa...@yahoo.in>.
Not sure if this is the right group to ask questions about flume:
I am getting an exception about unable to open a port in flume when trying to create a remote agent, more details below:
---------------------------
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
---------------------------
Here r the steps i have followed:
Step 1: Here is my agent3.conf created in the flume/conf dir:
******
agent3.sources = avrosource
agent3.sinks = filesink
agent3.channels = jdbcchannel
agent3.sources.avrosource.type = avro
agent3.sources.avrosource.bind = localhost
agent3.sources.avrosource.port = 4000
agent3.sources.avrosource.threads = 5
agent3.sinks.filesink.type = FILE_ROLL
agent3.sinks.filesink.sink.directory = /home/satish/work/apache-flume-1.3.1-bin/files
agent3.sinks.filesink.sink.rollInterval = 0
agent3.channels.jdbcchannel.type = jdbc
agent3.sources.avrosource.channels = jdbcchannel
agent3.sinks.filesink.channel = jdbcchannel
******
Step 2: Then i have saved it successfully and created a new test file like this:
Step 3: echo "Hello World" > /home/satish/message3
Step 4: Tried executing this command:
./flume-ng avro-client -H localhost -p 4000 -F /home/satish/message3
I get this exception below, please help:
------------------------------
Djava.library.path=:/home/satish/work/hadoop-1.0.4/libexec/../lib/native/Linux-i386-32 org.apache.flume.client.avro.AvroCLIClient -H ubuntu -p 41414 -F /usr/logs/log.10
13/05/20 04:55:30 ERROR avro.AvroCLIClient: Unable to open connection to Flume. Exception follows.
org.apache.flume.FlumeException: NettyAvroRpcClient { host: ubuntu, port: 41414 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
at org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
at org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
Caused by: java.io.IOException: Error connecting to ubuntu/127.0.0.1:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
... 5 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Please help.
Thanks
Sai
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Kai for confirming it.
________________________________
From: Kai Voigt <k...@123.org>
To: user@hadoop.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Saturday, 13 April 2013 7:18 AM
Subject: Re: 100K Maps scenario
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
>Just a follow up to see if anyone can shed some light on this:
>My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
>The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
>Will appreciate any thoughts on this.
>Thanks
>Sai
>
>________________________________
> From: Sai Sai <sa...@yahoo.in>
>To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 12 April 2013 1:37 PM
>Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
>
>
>Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
>Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
>Plesae pour some light.
>Thanks
>Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Kai Voigt <k...@123.org>.
No, only one copy of each block will be processed.
If a task fails, it will be retried on another copy. Also, if speculative execution is enabled, slow tasks might be executed twice in parallel. But this will only happen rarely.
Kai
Am 12.04.2013 um 18:45 schrieb Sai Sai <sa...@yahoo.in>:
>
> Just a follow up to see if anyone can shed some light on this:
> My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
> The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
> Will appreciate any thoughts on this.
> Thanks
> Sai
> From: Sai Sai <sa...@yahoo.in>
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
> Sent: Friday, 12 April 2013 1:37 PM
> Subject: Re: Does a Map task run 3 times on 3 TTs or just once
>
> Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
> Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
> Plesae pour some light.
> Thanks
> Sai
>
>
--
Kai Voigt
k@123.org
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Just a follow up to see if anyone can shed some light on this:
My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
Will appreciate any thoughts on this.
Thanks
Sai
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
Sent: Friday, 12 April 2013 1:37 PM
Subject: Re: Does a Map task run 3 times on 3 TTs or just once
Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
Plesae pour some light.
Thanks
Sai
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Just a follow up to see if anyone can shed some light on this:
My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
Will appreciate any thoughts on this.
Thanks
Sai
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
Sent: Friday, 12 April 2013 1:37 PM
Subject: Re: Does a Map task run 3 times on 3 TTs or just once
Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
Plesae pour some light.
Thanks
Sai
Re: 10 TB of a data file.
Posted by Sai Sai <sa...@yahoo.in>.
In real world can a file be of this big size as 10 TB?
Will the data be put into a txt file or what kind of a file?
If someone would like to open such a big file to look at the content will OS support opening such big files?
If not how to handle this kind of scenario?
Any input will be appreciated.
Thanks
Sai
Re: 10 TB of a data file.
Posted by Sai Sai <sa...@yahoo.in>.
In real world can a file be of this big size as 10 TB?
Will the data be put into a txt file or what kind of a file?
If someone would like to open such a big file to look at the content will OS support opening such big files?
If not how to handle this kind of scenario?
Any input will be appreciated.
Thanks
Sai
Re: 10 TB of a data file.
Posted by Sai Sai <sa...@yahoo.in>.
In real world can a file be of this big size as 10 TB?
Will the data be put into a txt file or what kind of a file?
If someone would like to open such a big file to look at the content will OS support opening such big files?
If not how to handle this kind of scenario?
Any input will be appreciated.
Thanks
Sai
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Just a follow up to see if anyone can shed some light on this:
My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
Will appreciate any thoughts on this.
Thanks
Sai
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
Sent: Friday, 12 April 2013 1:37 PM
Subject: Re: Does a Map task run 3 times on 3 TTs or just once
Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
Plesae pour some light.
Thanks
Sai
Re: 100K Maps scenario
Posted by Sai Sai <sa...@yahoo.in>.
Just a follow up to see if anyone can shed some light on this:
My understanding is that each block after getting replicated 3 times, a map task is run on each of the replica in parallel.
The thing i am trying to double verify is in a scenario where a file is split into 10K or 100K or more blocks it will result in atleast 300K Map tasks being performed and this looks like an overkill from a performance or just a logical perspective.
Will appreciate any thoughts on this.
Thanks
Sai
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Sai Sai <sa...@yahoo.in>
Sent: Friday, 12 April 2013 1:37 PM
Subject: Re: Does a Map task run 3 times on 3 TTs or just once
Just wondering if it is right to assume that a Map task is run 3 times on 3 different TTs in parallel and whoever completes processing the task first that output is picked up and written to intermediate location.
Or is it true that a map task even though its data is replicated 3 times will run only once and other 2 will be on the stand by just incase this fails the second one will run followed by 3rd one if the 2nd Mapper fails.
Plesae pour some light.
Thanks
Sai