Client JVM hangs with Flume1.6 log4j appender

[Hedi BOUFAIED <he...@amadeus.com>]

Hello everyone,

I am facing an issue with Flume 1.6 using the log4j flume appender and I am looking for some help.

The picture is:
[CLIENT JVM : LOG4J -> FLUME-LOG4J-APPENDER ]---> [FLUME JVM : AVRO-SRC -> MEM CHANNEL -> LOGGER SINK]

After all the logs are sent by the client's log4j appender - and received and processed correctly by the flume agent (using avro source) - the client JVM (which is a simple test) hangs forever and never exits.
When the client is recompiled and executed with the jars from Flume 1.5, it executes and terminates correctly...

An extract of the client execution with jdb seems to indicate that, with Flume 1.6 jars, after the main thread terminates, some threads related to Avro / NettyTransceiver  remain alive hung in some deadlock.
Has anyone hit this kind of issue ?
I am copying below the jdb session output, my client's java code and log4j.properties and flume.conf files.

Thanks in advance for any help you might provide !

Hedi


JDB SESSION:
---
$ jdb Main
Initializing jdb ...
> run
run Main
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
>
VM Started: Using default maxIOWorkers
Start of main
This is a 1st log message
This is a 2nd log message
This is the 3rd and last message
End of main
suspend
All threads suspended.
> threads
Group system:
  (java.lang.ref.Reference$ReferenceHandler)0x128 Reference Handler                    cond. waiting
  (java.lang.ref.Finalizer$FinalizerThread)0x127  Finalizer                            cond. waiting
  (java.lang.Thread)0x126                         Signal Dispatcher                    running
Group main:
  (java.lang.Thread)0x293                         New I/O  worker #1                   running
  (java.lang.Thread)0x29b                         New I/O  worker #2                   running
  (java.lang.Thread)0x29c                         New I/O  worker #3                   running
  (java.lang.Thread)0x29d                         New I/O  worker #4                   running
  (java.lang.Thread)0x29e                         New I/O  worker #5                   running
  (java.lang.Thread)0x29f                         New I/O  worker #6                   running
  (java.lang.Thread)0x2a0                         New I/O  worker #7                   running
  (java.lang.Thread)0x2a1                         New I/O  worker #8                   running
  (java.lang.Thread)0x31f                         Avro NettyTransceiver Boss 1         cond. waiting
  (java.lang.Thread)0x320                         Hashed wheel timer #1                sleeping
  (java.lang.Thread)0x4bd                         Flume Avro RPC Client Call Invoker 1 cond. waiting
  (java.lang.Thread)0x535                         Flume Avro RPC Client Call Invoker 2 cond. waiting
  (java.lang.Thread)0x536                         DestroyJavaVM                        running
---

Client source code and log4j.properties:
---
import org.apache.log4j.Logger;
class Main {
    final static Logger logger = Logger.getLogger(Main.class);
    public static void main(String[] args) {
        System.out.println("Start of main");
        logger.warn("This is a 1st log message");
        logger.warn("This is a 2nd log message");
        logger.warn("This is the 3rd and last log message");
        System.out.println("End of main");
    }
}
---
log4j.rootLogger=INFO, stdout, flume

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%m%n

log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppender
log4j.appender.flume.Port=5554
log4j.appender.flume.Hostname=localhost
log4j.appender.flume.layout=org.apache.log4j.PatternLayout
log4j.appender.flume.layout.ConversionPattern=%m/%n
---

Flume.conf file:
---
agent.sources = avroSource
agent.channels = logChannel
agent.sinks = loggerSink

agent.sources.avroSource.type = avro
agent.sources.avroSource.bind = 0.0.0.0
agent.sources.avroSource.port = 5554
agent.sources.avroSource.channels = logChannel

agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = logChannel

agent.channels.logChannel.type = memory
agent.channels.logChannel.capacity = 100
---
END