You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/08/16 11:49:54 UTC
[PR] [python] Avoid concurrent import failures during split deserialization [paimon]
XiaoHongbo-Hope opened a new pull request, #9250:
URL: https://github.com/apache/paimon/pull/9250
Ray workers may deserialize pickled split objects concurrently when many scan splits are scheduled. Importing a low-level pypaimon submodule currently executes the package root first, whose eager public exports load a broad catalog, filesystem, schema, and read dependency graph. Concurrent imports can then observe partially initialized modules or hit a Python module-lock deadlock.
This change lazily resolves the existing root-level public exports on Python 3.7 and newer, while retaining the eager compatibility path for Python 3.6. Public imports such as from pypaimon import CatalogFactory remain unchanged.
A fresh-process regression test concurrently imports and deserializes a real QueryAuthSplit containing DataFileMeta instances across 24 threads.
Validation:
- concurrent import regression repeated 30 times
- query-auth, catalog environment, and concurrent import tests: 61 passed
- flake8 and git diff --check passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Avoid concurrent import failures during split deserialization [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9250:
URL: https://github.com/apache/paimon/pull/9250#issuecomment-5311731305
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Avoid concurrent import failures during split deserialization [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9250:
URL: https://github.com/apache/paimon/pull/9250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org