You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "QuakeWang (via GitHub)" <gi...@apache.org> on 2026/06/04 14:34:42 UTC
[PR] [python][ray] Abort worker writes on failure [paimon]
QuakeWang opened a new pull request, #8124:
URL: https://github.com/apache/paimon/pull/8124
### Purpose
Ray write tasks previously closed on the worker-side `TableWrite` when write, prepare, or close failed. That is unsafe because close can flush pending data, and `prepare_commit()` can materialize normal, blob, or vector files before a later failure prevents the driver commit.
This PR makes worker-side Ray writes abort on failure and propagates abort through `TableWrite` and `FileStoreWrite`. It also keeps dedicated blob/vector metadata reachable from the parent writer so abort can delete files produced before the failed commit path.
### Tests
- `python -m pytest pypaimon/tests/ray_sink_test.py::RaySinkTest::test_write -q`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Abort worker writes on failure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8124:
URL: https://github.com/apache/paimon/pull/8124#issuecomment-4645712923
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Abort worker writes on failure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8124:
URL: https://github.com/apache/paimon/pull/8124#issuecomment-4637978749
I reviewed this PR as part of a pass over recent open non-draft Paimon PRs. I did not find any clear correctness, compatibility, or regression issue to flag from the current diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Abort worker writes on failure [paimon]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on PR #8124:
URL: https://github.com/apache/paimon/pull/8124#issuecomment-4628811514
@XiaoHongbo-Hope Thanks for catching this. You are right: `RayDatasink.write()` only handled close failures that propagated, while `DedicatedFormatWriter.close()` could abort and swallow the exception.
I fixed it by re-raising after abort in `DedicatedFormatWriter.close()`, and applied the same fix to `DataVectorWriter.close()` since it had the same pattern. I also added regressions for the Ray/Dedicated path and the vector writer path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Abort worker writes on failure [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #8124:
URL: https://github.com/apache/paimon/pull/8124#issuecomment-4628655866
Thanks for the PR! I agree abort is useful before prepare_commit() succeeds, but I’m worried about aborting after commit messages have already been produced.
The main risk here is after prepare_commit() succeeds. RayDatasink.write() will return the prepared commit messages after close(). But DedicatedFormatWriter.close() catches exceptions, calls abort(), and does not re-raise.
So if close() fails, abort() may delete files referenced by the prepared commit messages, while the worker still returns those messages to the driver. The driver may then commit a manifest with missing files.
We hit an abort-related data-safety issue in production before; see #7232 for context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Abort worker writes on failure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8124:
URL: https://github.com/apache/paimon/pull/8124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org