You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/07/19 10:38:30 UTC
[PR] [VL] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
malinjawi opened a new pull request, #12562:
URL: https://github.com/apache/gluten/pull/12562
## What changes are proposed in this pull request?
The columnar V2 write path commits every task attempt unconditionally — it ignores `BatchWrite.useCommitCoordinator`. With speculative execution or a retried stage, two attempts of the same partition can both call `DataWriter.commit()` and produce duplicate output.
This brings `WritingColumnarBatchSparkTask` in line with vanilla Spark's `DataWritingSparkTask`:
- Thread `useCommitCoordinator` from the `BatchWrite` into the task, and gate `commit()` behind `OutputCommitCoordinator.canCommit`, throwing the standard commit-denied error when the attempt loses.
- Skip `abort()` once a commit has succeeded. This path updates native write metrics after commit, and a failure there was aborting already-durable output. (Vanilla has no post-commit code that can throw, so it doesn't need this guard; we do.)
- Count rows as `Long` instead of `Int` to avoid overflow on large partitions.
## How was this patch tested?
New `ColumnarDataWritingSparkTaskSuite` drives the task with a recording writer factory and a stub commit coordinator: a denied attempt throws and aborts+closes the writer, a failure after commit does not abort, a failure before commit does, row counts accumulate past `Int.MaxValue`, and the coordinator-off path is unchanged.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (reviewed and tested before submission)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12562:
URL: https://github.com/apache/gluten/pull/12562#issuecomment-5015531641
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12562:
URL: https://github.com/apache/gluten/pull/12562#issuecomment-5239310043
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12562:
URL: https://github.com/apache/gluten/pull/12562#issuecomment-5015398113
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12562:
URL: https://github.com/apache/gluten/pull/12562#issuecomment-5986773182
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12562:
URL: https://github.com/apache/gluten/pull/12562#issuecomment-5825265374
This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12562:
URL: https://github.com/apache/gluten/pull/12562#issuecomment-5015508414
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [CORE] Honor BatchWrite.useCommitCoordinator in columnar V2 writes [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12562: [CORE] Honor BatchWrite.useCommitCoordinator in columnar V2 writes
URL: https://github.com/apache/gluten/pull/12562
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org