You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wudanzy (via GitHub)" <gi...@apache.org> on 2026/04/22 01:59:58 UTC

[I] [SPARK] V2 DataSource write path throws SparkException instead of CommitDeniedException, causing spurious stage failures with speculation [spark]

wudanzy opened a new issue, #55469:
URL: https://github.com/apache/spark/issues/55469

   ### Describe the bug
   
   When speculation is enabled, the `OutputCommitCoordinator` denies commit for losing speculative task attempts. The **V1 write path** (`SparkHadoopMapRedUtil.commitTask()`) correctly throws `CommitDeniedException`, which the executor converts to `TaskCommitDenied` with `countTowardsTaskFailures=false`. However, the **V2 write path** (`WritingSparkTask` in `WriteToDataSourceV2Exec.scala`) calls `QueryExecutionErrors.commitDeniedError()` which returns a **plain `SparkException`**. The executor does not recognize this as a commit denial, so each denied speculative attempt is counted as a real task failure. After `spark.task.maxFailures` (default 4) denials, the stage is incorrectly aborted.
   
   This affects **all V2 DataSource writes** with speculation enabled, including Structured Streaming jobs using `ForeachWriter`.
   
   ### V1 path (correct)
   ```scala
   // SparkHadoopMapRedUtil.scala:85
   throw new CommitDeniedException(message, ctx.stageId(), splitId, ctx.attemptNumber())
   // → Executor catches CommitDeniedException (Executor.scala:777)
   // → Converts to TaskCommitDenied (countTowardsTaskFailures=false) ✓
   ```
   
   ### V2 path (buggy)
   ```scala
   // WriteToDataSourceV2Exec.scala:590
   throw QueryExecutionErrors.commitDeniedError(partId, taskId, attemptId, stageId, stageAttempt)
   // → QueryExecutionErrors returns plain SparkException
   // → Executor generic Throwable handler (countTowardsTaskFailures=true) ✗
   ```
   
   ### How to Reproduce
   
   1. Run a Structured Streaming job with `ForeachWriter` (or any V2 DataSource sink)
   2. Enable speculation: `spark.speculation=true`
   3. Under resource pressure (e.g., container preemptions), Spark launches speculative task copies
   4. The `OutputCommitCoordinator` authorizes one attempt and denies the rest
   5. Each denial is counted as a task failure → stage aborts after `maxTaskFailures` denials
   
   ### Expected Behavior
   
   Commit denials for speculative task attempts should **not** count toward `maxTaskFailures`, matching V1 behavior. The stage should complete successfully as long as one attempt commits.
   
   ### Proposed Fix
   
   Change `WritingSparkTask.run()` in `WriteToDataSourceV2Exec.scala` to throw `CommitDeniedException` (from `spark-core`) directly, instead of calling `QueryExecutionErrors.commitDeniedError()` which returns a plain `SparkException`. This ensures the executor recognizes the denial and reports `TaskCommitDenied` with `countTowardsTaskFailures=false`.
   
   ### Environment
   
   - Spark version: 3.5.3, master (also present in branch-4.0)
   - Observed in production Structured Streaming job with ForeachWriter on YARN with speculation enabled


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [I] [SPARK-56564] V2 DataSource write path throws SparkException instead of CommitDeniedException, causing spurious stage failures with speculation [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #55469: [SPARK-56564] V2 DataSource write path throws SparkException instead of CommitDeniedException, causing spurious stage failures with speculation
URL: https://github.com/apache/spark/issues/55469


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org