You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Sushruth Puttaswamy <su...@nor1.com> on 2011/12/14 20:29:18 UTC
flume collector throws Could not create serversocket on address
0.0.0.0/0.0.0.0:35862
I have a simple distributed flume setup as below..
Installed flume node on a server A. Server A has flume agent node running.
Installed flume master ( 0.9.4 ) on one of the nodes B in our amazon hadoop cluster. It has cloudera's version 3 with update 2.
modified the flume configuration on server A to point to node B on amazon.
Started flume node on server A.
Started flume master on node B on amazon.
Started a collector C on the same node B on amazon.
I then issued the following configurations to transfer 1 single file from node A to the collector on node B & then write to local file system.
A:text("/etc/services") | agentSink("B",35862)
C:collectorSource(35862) | collectorSink("file:///tmp/flume/collected","raw")
I get the following error in the collector log…
2011-12-14 19:05:21,410 [logicalNode collector2-22] INFO thrift.ThriftEventSource: Starting blocking thread pool server on port 35862...
2011-12-14 19:05:21,412 [logicalNode collector2-22] ERROR server.TSaneThreadPoolServer: Error occurred during listening.
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:35862.
at org.apache.thrift.transport.TSaneServerSocket.bind(TSaneServerSocket.java:110)
at org.apache.thrift.transport.TSaneServerSocket.listen(TSaneServerSocket.java:116)
at org.apache.thrift.server.TSaneThreadPoolServer.start(TSaneThreadPoolServer.java:162)
at com.cloudera.flume.handlers.thrift.ThriftEventSource.open(ThriftEventSource.java:151)
at com.cloudera.flume.collector.CollectorSource.open(CollectorSource.java:67)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:87)
2011-12-14 19:05:21,444 [logicalNode collector2-22] INFO debug.ConsoleEventSink: ConsoleEventSink( debug ) opened
The collector, agent node & the master are running fine on their respective ports. When I do a netstat on my collector & grep for its port I get multiple entries. Is the port occupied already?
Is this issue similar to https://issues.cloudera.org/browse/FLUME-706. If so, where can I get a patch in the form of a jar file for flume?
Sushruth
flume agent throws connection refused error on port 35853
Posted by Sushruth Puttaswamy <su...@nor1.com>.
I was able to get around my earlier problem. Now I have a simple config to tail a log file & then write it onto a local file system on a node within amazon's hadoop cluster.
My config is
agent: tailDir("/tmp/test") | agentSink("B", 35853)
collector: collectorSource(35853) | collectorSink("file:///tmp/collected", "raw")
Where "B" is my node on amazon which has both the master & "collector" running.
/tmp/test directory has 2 log files.
I initially get the following error on the agent….
2011-12-15 20:50:09,530 INFO com.cloudera.flume.handlers.debug.InsistentOpenDecorator: open attempt 3 failed, backoff (8000ms): Failed to open thrift event sink to ec2-184-73-130-138.compute-1.amazonaws.com:35853 : java.net.ConnectException: Connection refused
After a bunch of these errors I see a repeated sequence of open & closing of seq files. But I don't really see anything on the collector side, no files transferred.
2011-12-15 21:03:59,013 INFO com.cloudera.flume.handlers.hdfs.SeqfileEventSink: constructed new seqfile event sink: file=/tmp/flume-flume/agent/A/writing/20111215-210359013+0000.55048298201816823.00000023
2011-12-15 21:03:59,014 INFO com.cloudera.flume.handlers.endtoend.AckListener$Empty: Empty Ack Listener began 20111215-210359013+0000.55048298201816823.00000023
2011-12-15 21:04:09,094 INFO com.cloudera.flume.handlers.hdfs.SeqfileEventSink: closed /tmp/flume-flume/agent/A/writing/20111215-210359013+0000.55048298201816823.00000023
2011-12-15 21:04:09,094 INFO com.cloudera.flume.handlers.endtoend.AckListener$Empty: Empty Ack Listener ended 20111215-210359013+0000.55048298201816823.00000023
2011-12-15 21:04:09,094 INFO com.cloudera.flume.agent.durability.NaiveFileWALManager: File lives in /tmp/flume-flume/agent/A/writing/20111215-210359013+0000.55048298201816823.00000023
The collector node is up & running & port 35853 is indeed accesible.
Any thoughts on this?
Sushruth Puttaswamy
From: Sushruth Puttaswamy <su...@nor1.com>>
Reply-To: "flume-user@incubator.apache.org<ma...@incubator.apache.org>" <fl...@incubator.apache.org>>
Date: Wed, 14 Dec 2011 13:29:18 -0600
To: "flume-user@incubator.apache.org<ma...@incubator.apache.org>" <fl...@incubator.apache.org>>
Subject: flume collector throws Could not create serversocket on address 0.0.0.0/0.0.0.0:35862
I have a simple distributed flume setup as below..
Installed flume node on a server A. Server A has flume agent node running.
Installed flume master ( 0.9.4 ) on one of the nodes B in our amazon hadoop cluster. It has cloudera's version 3 with update 2.
modified the flume configuration on server A to point to node B on amazon.
Started flume node on server A.
Started flume master on node B on amazon.
Started a collector C on the same node B on amazon.
I then issued the following configurations to transfer 1 single file from node A to the collector on node B & then write to local file system.
A:text("/etc/services") | agentSink("B",35862)
C:collectorSource(35862) | collectorSink("file:///tmp/flume/collected","raw")
I get the following error in the collector log…
2011-12-14 19:05:21,410 [logicalNode collector2-22] INFO thrift.ThriftEventSource: Starting blocking thread pool server on port 35862...
2011-12-14 19:05:21,412 [logicalNode collector2-22] ERROR server.TSaneThreadPoolServer: Error occurred during listening.
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:35862.
at org.apache.thrift.transport.TSaneServerSocket.bind(TSaneServerSocket.java:110)
at org.apache.thrift.transport.TSaneServerSocket.listen(TSaneServerSocket.java:116)
at org.apache.thrift.server.TSaneThreadPoolServer.start(TSaneThreadPoolServer.java:162)
at com.cloudera.flume.handlers.thrift.ThriftEventSource.open(ThriftEventSource.java:151)
at com.cloudera.flume.collector.CollectorSource.open(CollectorSource.java:67)
at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:87)
2011-12-14 19:05:21,444 [logicalNode collector2-22] INFO debug.ConsoleEventSink: ConsoleEventSink( debug ) opened
The collector, agent node & the master are running fine on their respective ports. When I do a netstat on my collector & grep for its port I get multiple entries. Is the port occupied already?
Is this issue similar to https://issues.cloudera.org/browse/FLUME-706. If so, where can I get a patch in the form of a jar file for flume?
Sushruth