You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by wang zheng <wa...@gmail.com> on 2018/08/03 05:08:11 UTC
Review Request 68179: Null point using hdfs sink when flume startup
faster than hdfs
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68179/
-----------------------------------------------------------
Review request for Flume.
Repository: flume-git
Description
-------
When flume starts up faster than hdfs, then hdfs sink will report NullPoint, nothing will sink to hdfs. Patch have been tested in my dev env.
flume_1 | 2018-07-25 03:49:06,147 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:582)] Caught IOException writing to HDFSWriter (opt). Closing file (hdfs://namenode:9000/flume/log00/201807/FlumeData.1532490542456.avro.tmp) and rethrowing exception.
flume_1 | 2018-07-25 03:49:06,161 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:393)] Closing hdfs://namenode:9000/flume/log00/201807/FlumeData.1532490542456.avro.tmp
flume_1 | 2018-07-25 03:49:06,164 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
flume_1 | java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
flume_1 | at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
flume_1 | at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
flume_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
flume_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
flume_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
flume_1 | at java.lang.Thread.run(Thread.java:748)
flume_1 | 2018-07-25 03:49:06,164 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
flume_1 | org.apache.flume.EventDeliveryException: java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
flume_1 | at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
flume_1 | at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
flume_1 | at java.lang.Thread.run(Thread.java:748)
flume_1 | Caused by: java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
flume_1 | at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AvroEventSerializer.java 32317424
Diff: https://reviews.apache.org/r/68179/diff/1/
Testing
-------
Thanks,
wang zheng
Re: Review Request 68179: Null point using hdfs sink when flume
startup faster than hdfs
Posted by wang zheng <wa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68179/
-----------------------------------------------------------
(Updated 八月 3, 2018, 5:09 a.m.)
Review request for Flume.
Repository: flume-git
Description (updated)
-------
JIRA https://issues.apache.org/jira/browse/FLUME-3264
When flume starts up faster than hdfs, then hdfs sink will report NullPoint, nothing will sink to hdfs. Patch have been tested in my dev env.
flume_1 | 2018-07-25 03:49:06,147 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:582)] Caught IOException writing to HDFSWriter (opt). Closing file (hdfs://namenode:9000/flume/log00/201807/FlumeData.1532490542456.avro.tmp) and rethrowing exception.
flume_1 | 2018-07-25 03:49:06,161 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:393)] Closing hdfs://namenode:9000/flume/log00/201807/FlumeData.1532490542456.avro.tmp
flume_1 | 2018-07-25 03:49:06,164 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
flume_1 | java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
flume_1 | at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
flume_1 | at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
flume_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
flume_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
flume_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
flume_1 | at java.lang.Thread.run(Thread.java:748)
flume_1 | 2018-07-25 03:49:06,164 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
flume_1 | org.apache.flume.EventDeliveryException: java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
flume_1 | at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
flume_1 | at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
flume_1 | at java.lang.Thread.run(Thread.java:748)
flume_1 | Caused by: java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
flume_1 | at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
Diffs
-----
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AvroEventSerializer.java 32317424
Diff: https://reviews.apache.org/r/68179/diff/1/
Testing
-------
Thanks,
wang zheng