ElasticSearch Sink stopped due to FileChannel error? Or other way around?

[Matt Wise <ma...@nextdoor.com>]

I'm troubleshooting an error we saw where our ElasticSearch
(REST-configured) sink stopped sucking events out of the file channel. I'm
not sure how to read this exception -- was there an error in the file
channel that caused the sink to bail, or was it the other way around?
>
>
> 09 Jul 2015 15:33:36,143 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.elasticsearch.ElasticSearchSink.process:221)  -
> Failed to commit transaction. Transaction rolled back.
> org.apache.flume.ChannelException: Commit failed due to IO error
> [channel=fc-structured-es]
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:582)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:207)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:380)
> at
> org.apache.flume.channel.file.LogFileV3.writeDelimitedTo(LogFileV3.java:148)
> at
> org.apache.flume.channel.file.LogFileV3$Writer.<init>(LogFileV3.java:209)
> at
> org.apache.flume.channel.file.LogFileFactory.getWriter(LogFileFactory.java:77)
> at org.apache.flume.channel.file.Log.roll(Log.java:946)
> at org.apache.flume.channel.file.Log.roll(Log.java:915)
> at org.apache.flume.channel.file.Log.commit(Log.java:894)
> at org.apache.flume.channel.file.Log.commitTake(Log.java:766)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:578)
> ... 5 more
> 09 Jul 2015 15:33:36,143 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
> event. Exception follows.
> org.apache.flume.ChannelException: Commit failed due to IO error
> [channel=fc-structured-es]
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:582)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:207)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:380)
> at
> org.apache.flume.channel.file.LogFileV3.writeDelimitedTo(LogFileV3.java:148)
> at
> org.apache.flume.channel.file.LogFileV3$Writer.<init>(LogFileV3.java:209)
> at
> org.apache.flume.channel.file.LogFileFactory.getWriter(LogFileFactory.java:77)
> at org.apache.flume.channel.file.Log.roll(Log.java:946)
> at org.apache.flume.channel.file.Log.roll(Log.java:915)
> at org.apache.flume.channel.file.Log.commit(Log.java:894)
> at org.apache.flume.channel.file.Log.commitTake(Log.java:766)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:578)
> ... 5 more
> 09 Jul 2015 15:33:36,143 INFO  [conf-file-poller-0]
> (org.apache.flume.sink.elasticsearch.ElasticSearchSink.stop:376)  -
> ElasticSearch sink {} stopping
> 09 Jul 2015 15:33:36,144 INFO  [conf-file-poller-0]
> (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:150)  -
> Component type: SINK, name: esstructured stopped


Matt Wise
Sr. Systems Architect
Nextdoor.com