You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anika Kelhanka (Jira)" <ji...@apache.org> on 2026/10/07 05:43:00 UTC
[jira] [Updated] (SPARK-60023) FileFormatWriter throws NPE in setupJob with ManifestCommitter
[ https://issues.apache.org/jira/browse/SPARK-60023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anika Kelhanka updated SPARK-60023:
-----------------------------------
Description:
SPARK-56919 moved `committer.setupJob(job)` ahead of `materializeAdaptiveSparkPlan(plan)` in `FileFormatWriter.write` so that a failure during AQE materialization triggers `committer.abortJob(job)` instead of leaving an un-setup job that deletes the output directory.
However, `job.getConfiguration.set("spark.sql.sources.writeJobUUID", description.uuid)` remained inside the try block after `materializeAdaptiveSparkPlan(plan)`.
Committers that read `spark.sql.sources.writeJobUUID` during `setupJob` or `abortJob`—such as Hadoop's ManifestCommitter (the default committer for `gs://` and `abfs://` since Hadoop 3.4)—fall back to `jobContext.getJobID().toString()` when `spark.sql.sources.writeJobUUID` is unset, throwing a `NullPointerException` because `job.getJobID()` is null for a Job that was never submitted:
{code:java}
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}
Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling committer.setupJob(job) resolves the issue.
was:
SPARK-56919 (commit 802b0b0d7e8c) moved committer.setupJob(job) ahead of materializeAdaptiveSparkPlan(plan) in FileFormatWriter.write so that a failure during AQE materialization triggers committer.abortJob(job) instead of leaving an un-setup job that deletes the output directory.
However, job.getConfiguration.set("spark.sql.sources.writeJobUUID", description.uuid) remained inside the try block after materializeAdaptiveSparkPlan(plan).
Committers that read spark.sql.sources.writeJobUUID during setupJob or abortJob—such as Hadoop's ManifestCommitter (the default committer for gs:// and abfs:// since Hadoop 3.4)—fall back to jobContext.getJobID().toString() when spark.sql.sources.writeJobUUID is unset, throwing a NullPointerException because job.getJobID() is null for a Job that was never submitted:
{code}
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}
Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling committer.setupJob(job) resolves the issue.
> FileFormatWriter throws NPE in setupJob with ManifestCommitter
> -----------------------------------------------------------------
>
> Key: SPARK-60023
> URL: https://issues.apache.org/jira/browse/SPARK-60023
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Anika Kelhanka
> Priority: Major
>
> SPARK-56919 moved `committer.setupJob(job)` ahead of `materializeAdaptiveSparkPlan(plan)` in `FileFormatWriter.write` so that a failure during AQE materialization triggers `committer.abortJob(job)` instead of leaving an un-setup job that deletes the output directory.
> However, `job.getConfiguration.set("spark.sql.sources.writeJobUUID", description.uuid)` remained inside the try block after `materializeAdaptiveSparkPlan(plan)`.
> Committers that read `spark.sql.sources.writeJobUUID` during `setupJob` or `abortJob`—such as Hadoop's ManifestCommitter (the default committer for `gs://` and `abfs://` since Hadoop 3.4)—fall back to `jobContext.getJobID().toString()` when `spark.sql.sources.writeJobUUID` is unset, throwing a `NullPointerException` because `job.getJobID()` is null for a Job that was never submitted:
> {code:java}
> java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
> at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
> at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
> at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
> at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
> at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
> at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
> {code}
> Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling committer.setupJob(job) resolves the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org