[jira] [Created] (FLUME-3326) HDFSEventSink should log the exceptions before rolling back

["Miklos Szurap (JIRA)" <ji...@apache.org>]

Miklos Szurap created FLUME-3326:
------------------------------------

             Summary: HDFSEventSink should log the exceptions before rolling back
                 Key: FLUME-3326
                 URL: https://issues.apache.org/jira/browse/FLUME-3326
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
            Reporter: Miklos Szurap


Currently in the {{HDFSEventSink}} if the [transaction.commit()|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L444] fails then in the corresponding catch blocks for IOException and Throwable it tries to roll back the transaction with {{transaction.rollback()}} and logs the exception/throwable only after the rollback.
 However the rollback also can fail (for example with [FileChannel|https://github.com/apache/flume/blob/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java#L666] in form of a ChannelException which is a RuntimeException) throwing the exception one layer up and *hiding away the original cause*.
 Flume should first log the error, then try the rollback.
 In such cases we see just that the rollback failed. In a recent incident we saw it with free disk space issues, however this can be more general.
{noformat}
2018-11-14 06:21:12,400 ERROR org.apache.flume.source.SyslogUDPSource: Error writting to channel
org.apache.flume.ChannelException: Commit failed due to IO error [channel=my-channel]
at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:660)
at org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:272)
at org.apache.flume.source.SyslogUDPSource$syslogHandler.messageReceived(SyslogUDPSource.java:90)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.socket.oio.OioDatagramWorker.process(OioDatagramWorker.java:52)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Usable space exhausted, only 467148800 bytes remaining, required 524288026 bytes
at org.apache.flume.channel.file.Log.rollback(Log.java:744)
at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:658)
... 14 more 
{noformat}
(line numbers in the above may differ due to different flume version)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org