You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/04/13 10:29:34 UTC
[GitHub] [iceberg] dixingxing0 opened a new issue #2470: NPE while closing OrcFileAppender
dixingxing0 opened a new issue #2470:
URL: https://github.com/apache/iceberg/issues/2470
We've met an NullPointerException in our flink job, has anyone met this before?
It seems like `rawWriter` is null which means `close` method was already invoked:
https://github.com/apache/orc/blob/22828f79a526069d9629719c9476b7addad91ae6/java/core/src/java/org/apache/orc/impl/PhysicalFsWriter.java#L416
https://github.com/apache/orc/blob/22828f79a526069d9629719c9476b7addad91ae6/java/core/src/java/org/apache/orc/impl/PhysicalFsWriter.java#L486
```java
java.io.IOException: Could not perform checkpoint 1011 for operator SourceConversion(table=[runtime-catalog.fdm_realtime.fdm_realtime_rcm_feature, source: xxxxxxxxx -> IcebergStreamWriter (15/160).
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:892) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:113) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:137) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:93) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:158) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:351) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:566) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:536) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_92]
Caused by: java.lang.NullPointerException
at org.apache.iceberg.shaded.org.apache.orc.impl.PhysicalFsWriter.writeFileMetadata(PhysicalFsWriter.java:416) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.shaded.org.apache.orc.impl.WriterImpl.writeMetadata(WriterImpl.java:571) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.shaded.org.apache.orc.impl.WriterImpl.writeFooter(WriterImpl.java:636) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.shaded.org.apache.orc.impl.WriterImpl.close(WriterImpl.java:728) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.orc.OrcFileAppender.close(OrcFileAppender.java:127) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.io.DataWriter.close(DataWriter.java:63) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.closeCurrent(BaseTaskWriter.java:283) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.close(BaseTaskWriter.java:299) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.io.PartitionedFanoutWriter.close(PartitionedFanoutWriter.java:65) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.io.BaseTaskWriter.complete(BaseTaskWriter.java:84) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.iceberg.flink.sink.IcebergStreamWriter.prepareSnapshotPreBarrier(IcebergStreamWriter.java:65) ~[iceberg-flink-runtime-0.11.0-auto.jar-1617266335824.jar-1617267644514.jar:?]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.prepareSnapshotPreBarrier(OperatorChain.java:266) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:249) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$8(StreamTask.java:921) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:911) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:879) ~[flink-dist_2.11-1.11.2-auto1.1-SNAPSHOT.jar:1.11.2-auto1.1-SNAPSHOT]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx commented on issue #2470: NPE while closing OrcFileAppender
Posted by GitBox <gi...@apache.org>.
openinx commented on issue #2470:
URL: https://github.com/apache/iceberg/issues/2470#issuecomment-818683316
In this line, we will reset the `currentWriter` to be null after closed the `currentWriter`. So in theory we won't close the `currentWriter` in apache iceberg layer. Is possible that the Orc writer did some logics to close the writer ? Or are there any approached to reproduce this bug ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dixingxing0 commented on issue #2470: NPE while closing OrcFileAppender
Posted by GitBox <gi...@apache.org>.
dixingxing0 commented on issue #2470:
URL: https://github.com/apache/iceberg/issues/2470#issuecomment-818727286
We can't reproduce this bug now, we've already switched the data format to `parquet`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] openinx edited a comment on issue #2470: NPE while closing OrcFileAppender
Posted by GitBox <gi...@apache.org>.
openinx edited a comment on issue #2470:
URL: https://github.com/apache/iceberg/issues/2470#issuecomment-818683316
In this line, we will reset the `currentWriter` to be null after closed the `currentWriter`. So in theory we won't close the `currentWriter` in apache iceberg layer. Is possible that the Orc writer did some logics to close the writer ? Or are there any approaches to reproduce this bug ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org