You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "snmvaughan (via GitHub)" <gi...@apache.org> on 2026/10/20 15:00:24 UTC
[PR] [SPARK-56720][4.2][SS] Fail subsequent async log writes after a prior failure in async progress tracking [spark]
snmvaughan opened a new pull request, #59642:
URL: https://github.com/apache/spark/pull/59642
### What changes were proposed in this pull request?
Backport of #55676 to branch-4.2. The cherry-pick is clean.
`AsyncOffsetSeqLog` and `AsyncCommitLog` now share the query's `ErrorNotifier`. After a write has failed:
- Offset and commit log write tasks that run later fail with the first error and don't write to the checkpoint.
- `ErrorNotifier.markError` keeps the first error and attaches later ones to it as suppressed exceptions, so a cascading failure can't replace the root cause.
### Why are the changes needed?
With async progress tracking, offset and commit log entries are written by a single-threaded executor. When the write of `offsets/N` fails, a `commits/N` write queued after it can still succeed. That leaves a commit log entry without its offset log entry in the checkpoint. The query can also fail with a later, cascading error instead of the I/O error that started it. The same gap exists on branch-4.2.
SPARK-60264 (#59518), which is already on branch-4.2, makes `AbortableStreamBasedCheckpointFileManager` report failed writes instead of only logging them. With async progress tracking, a commit log write queued behind such a failure can still go through without this change.
### Does this PR introduce _any_ user-facing change?
No. As in #55676, a query that hits a write failure still stops with a `StreamingQueryException`, which now reports the original error.
### How was this patch tested?
Includes the regression test from #55676, "async log writes record first error and gate subsequent writes", and its assertion change in `testAsyncWriteErrorsPermissionsIssue`, both in `AsyncProgressTrackingMicroBatchExecutionSuite`.
The changed main and test sources compile against the Spark 4.2.0 release jars with this branch's scalac options, and pass scalastyle. The suite was not run locally; this PR's CI runs it.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56720][4.2][SS] Fail subsequent async log writes after a prior failure in async progress tracking [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #59642:
URL: https://github.com/apache/spark/pull/59642#issuecomment-6300175672
**Merge Summary:**
- merged into branch-4.2 https://github.com/apache/spark/commit/e4be8def6cdfc3f05a369c08cace63df953d874f
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56720][4.2][SS] Fail subsequent async log writes after a prior failure in async progress tracking [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #59642: [SPARK-56720][4.2][SS] Fail subsequent async log writes after a prior failure in async progress tracking
URL: https://github.com/apache/spark/pull/59642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org