You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/08/11 17:32:00 UTC
[jira] [Created] (HADOOP-18402) S3A committer NPE in spark job abort
Steve Loughran created HADOOP-18402:
---------------------------------------
Summary: S3A committer NPE in spark job abort
Key: HADOOP-18402
URL: https://issues.apache.org/jira/browse/HADOOP-18402
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.3.9
Reporter: Steve Loughran
Assignee: Steve Loughran
NPE happening in spark {{HadoopMapReduceCommitProtocol.abortJob}} when jobID is null
{code}
- save()/findClass() - non-partitioned table - Overwrite *** FAILED ***
java.lang.NullPointerException:
at org.apache.hadoop.fs.s3a.commit.impl.CommitContext.<init>(CommitContext.java:159)
at org.apache.hadoop.fs.s3a.commit.impl.CommitOperations.createCommitContext(CommitOperations.java:652)
at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.initiateJobOperation(AbstractS3ACommitter.java:856)
at org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter.abortJob(AbstractS3ACommitter.java:909)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.abortJob(HadoopMapReduceCommitProtocol.scala:252)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:268)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:191)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
...
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org