You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2026/06/06 11:20:22 UTC
[PR] [hive] Bug fix for stuck thread on hive prepareCommit failure [paimon]
ArnavBalyan opened a new pull request, #8142:
URL: https://github.com/apache/paimon/pull/8142
### Purpose
- Today, any hive job writing to Paimon table failing before precommit, causes job to be permanently stuck and does not exit cleanly.
- This is because Paimon throws another exception in the cleanup process when there are no files to be cleaned up. Paimon throws unchecked runtime exception not caught at caller.
- This causes the parent thread to be permanently stuck waiting job failure to arrive (which does not happen since the runtime exception was not caught by hadoop).
- Ensure that we don't do unprotected access, and can handle exceptions when there are no precommit files produced.
- This also fixes an issue of CI being stuck and timing out in some cases when the UT has a failure.
### Tests
- UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hive] Bug fix for stuck thread on hive prepareCommit failure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8142:
URL: https://github.com/apache/paimon/pull/8142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hive] Bug fix for stuck thread on hive prepareCommit failure [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8142:
URL: https://github.com/apache/paimon/pull/8142#issuecomment-4642419778
Thanks for fixing the abort cleanup path. I think this needs one more guard before merge.
`readPreCommitFile` now returns `Collections.emptyList()` when a `.preCommit` file is missing, but the same `getAllPreCommitMessage(...)` path is used by both `abortJob` and successful `commitJob`. That tolerance is safe for abort cleanup, but unsafe for commit: if one task's `.preCommit` file is missing during `commitJob`, the job will commit only the remaining task messages and then delete the temp directory, which can produce a partial table commit instead of failing atomically.
Could we keep the missing-file tolerance only for the abort path? For example, pass an `ignoreMissing` flag into `getAllPreCommitMessage` / `readPreCommitFile`; `abortJob` can use tolerant mode, while `commitJob` should still fail if any expected pre-commit file is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hive] Bug fix for stuck thread on hive prepareCommit failure [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #8142:
URL: https://github.com/apache/paimon/pull/8142#issuecomment-4646996248
> Thanks for fixing the abort cleanup path. I think this needs one more guard before merge.
>
> `readPreCommitFile` now returns `Collections.emptyList()` when a `.preCommit` file is missing, but the same `getAllPreCommitMessage(...)` path is used by both `abortJob` and successful `commitJob`. That tolerance is safe for abort cleanup, but unsafe for commit: if one task's `.preCommit` file is missing during `commitJob`, the job will commit only the remaining task messages and then delete the temp directory, which can produce a partial table commit instead of failing atomically.
>
> Could we keep the missing-file tolerance only for the abort path? For example, pass an `ignoreMissing` flag into `getAllPreCommitMessage` / `readPreCommitFile`; `abortJob` can use tolerant mode, while `commitJob` should still fail if any expected pre-commit file is absent.
Thanks for the detailed review! Addressed all comments :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [hive] Bug fix for stuck thread on hive prepareCommit failure [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #8142:
URL: https://github.com/apache/paimon/pull/8142#issuecomment-4638352800
cc @JingsongLi @leaves12138 could you PTAL thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org