RE: NettyTransceiver Log Level (was: Avro Client Hangs after Connect / Timeout Waiting for Handshake FLUME-1329)

[Elmar Weber <El...@holidaycheck.com>]

Hello again,

I found the mistake, the problem was that in my log4j configuration file I did not set the level of org.apache.avro.ipc.NettyTransceiver to WARN. Hence the transceiver that sends the log messages tried to used itself for logging and probably blocked itself somewhere. I've seen that the default log4j configuration of Flume does this too.

Is this something that should be documented or checked by the sink during start? 

Thanks & Best regards,
Elmar 

-----Original Message-----
From: Elmar Weber [mailto:Elmar.Weber@holidaycheck.com] 
Sent: Montag, 6. August 2012 11:45
To: user@flume.apache.org
Subject: Avro Client Hangs after Connect / Timeout Waiting for Handshake FLUME-1329

Hello,

I'm currently trying to setup an Avro connection with the latest Flume NG sources to test the log4j Appender. The setup and start of the agent works flawlessly, the sink is set up as expected. The Avro client establishes the connection successfully (see below from server log):

12/08/06 05:28:07 INFO ipc.NettyServer: [id: 0x427eb6e2, /127.0.0.1:37660 => /127.0.0.1:41414] OPEN
12/08/06 05:28:07 INFO ipc.NettyServer: [id: 0x427eb6e2, /127.0.0.1:37660 => /127.0.0.1:41414] BOUND: /127.0.0.1:41414
12/08/06 05:28:07 INFO ipc.NettyServer: [id: 0x427eb6e2, /127.0.0.1:37660 => /127.0.0.1:41414] CONNECTED: /127.0.0.1:37660

However, when the client connects, it fails after 20 seconds with a Timeout Exception during the handshake:

 (java.lang.String) org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 41414 }: Failed to send event
	at org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:171)
	at org.apache.flume.clients.log4jappender.Log4jAppender.append(Log4jAppender.java:128)
[...]
Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient { host: localhost, port: 41414 }: Handshake timed out after 20000 ms
	at org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:204)
	at org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:159)
	... 23 more
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
	at java.util.concurrent.FutureTask.get(FutureTask.java:91)
	at org.apache.flume.api.NettyAvroRpcClient.append(NettyAvroRpcClient.java:202)
	... 24 more

>From the description it looks like https://issues.apache.org/jira/browse/FLUME-1329, but the changes from that issue are in the used version.

I tried both Windows 7 64bit (Flume 1.3.0-SNAPSHOT) and a Red Hat 64 Bit (Cloudera CDH4 with Flume NG 1.1.0) with JDK 1.6.31, neither did work.
Normal Flume NG operations work, e.g. read from file and write to HDFS.

Any tips how to further debug this or the possible source of the problem?

Thanks & Regards
Elmar