You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Anika Kelhanka (Jira)" <ji...@apache.org> on 2026/10/07 05:43:00 UTC

[jira] [Updated] (SPARK-60023) FileFormatWriter throws NPE in setupJob with ManifestCommitter

     [ https://issues.apache.org/jira/browse/SPARK-60023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anika Kelhanka updated SPARK-60023:
-----------------------------------
    Description: 
SPARK-56919 moved `committer.setupJob(job)` ahead of `materializeAdaptiveSparkPlan(plan)` in `FileFormatWriter.write` so that a failure during AQE materialization triggers `committer.abortJob(job)` instead of leaving an un-setup job that deletes the output directory.

However, `job.getConfiguration.set("spark.sql.sources.writeJobUUID", description.uuid)` remained inside the try block after `materializeAdaptiveSparkPlan(plan)`.

Committers that read `spark.sql.sources.writeJobUUID` during `setupJob` or `abortJob`—such as Hadoop's ManifestCommitter (the default committer for `gs://` and `abfs://` since Hadoop 3.4)—fall back to `jobContext.getJobID().toString()` when `spark.sql.sources.writeJobUUID` is unset, throwing a `NullPointerException` because `job.getJobID()` is null for a Job that was never submitted:
{code:java}
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
  at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}
Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling committer.setupJob(job) resolves the issue.

  was:
SPARK-56919 (commit 802b0b0d7e8c) moved committer.setupJob(job) ahead of materializeAdaptiveSparkPlan(plan) in FileFormatWriter.write so that a failure during AQE materialization triggers committer.abortJob(job) instead of leaving an un-setup job that deletes the output directory.

However, job.getConfiguration.set("spark.sql.sources.writeJobUUID", description.uuid) remained inside the try block after materializeAdaptiveSparkPlan(plan).

Committers that read spark.sql.sources.writeJobUUID during setupJob or abortJob—such as Hadoop's ManifestCommitter (the default committer for gs:// and abfs:// since Hadoop 3.4)—fall back to jobContext.getJobID().toString() when spark.sql.sources.writeJobUUID is unset, throwing a NullPointerException because job.getJobID() is null for a Job that was never submitted:

{code}
java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
  at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
  at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
  at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
{code}

Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling committer.setupJob(job) resolves the issue.


> FileFormatWriter throws NPE in setupJob with ManifestCommitter   
> -----------------------------------------------------------------
>
>                 Key: SPARK-60023
>                 URL: https://issues.apache.org/jira/browse/SPARK-60023
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Anika Kelhanka
>            Priority: Major
>
> SPARK-56919 moved `committer.setupJob(job)` ahead of `materializeAdaptiveSparkPlan(plan)` in `FileFormatWriter.write` so that a failure during AQE materialization triggers `committer.abortJob(job)` instead of leaving an un-setup job that deletes the output directory.
> However, `job.getConfiguration.set("spark.sql.sources.writeJobUUID", description.uuid)` remained inside the try block after `materializeAdaptiveSparkPlan(plan)`.
> Committers that read `spark.sql.sources.writeJobUUID` during `setupJob` or `abortJob`—such as Hadoop's ManifestCommitter (the default committer for `gs://` and `abfs://` since Hadoop 3.4)—fall back to `jobContext.getJobID().toString()` when `spark.sql.sources.writeJobUUID` is unset, throwing a `NullPointerException` because `job.getJobID()` is null for a Job that was never submitted:
> {code:java}
> java.lang.NullPointerException: Cannot invoke "org.apache.hadoop.mapreduce.JobID.toString()" because "jobId" is null
>   at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestCommitterSupport.buildJobUUID(ManifestCommitterSupport.java:118)
>   at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterConfig.<init>(ManifestCommitterConfig.java:190)
>   at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.enterCommitter(ManifestCommitter.java:181)
>   at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.setupJob(ManifestCommitter.java:196)
>   at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:198)
>   at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:165)
> {code}
> Setting spark.sql.sources.writeJobUUID on job.getConfiguration before calling committer.setupJob(job) resolves the issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org