You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "linrrzqqq (via GitHub)" <gi...@apache.org> on 2026/04/07 09:26:11 UTC
[PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
linrrzqqq opened a new pull request, #62167:
URL: https://github.com/apache/doris/pull/62167
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: https://github.com/apache/doris/pull/61280
Problem Summary:
When two imported modules share the same top-level package name, python_server imports them package by package, and clean up `sys.modules` after import, which causes incorrect lock key assignment and leads to thread contention or interference between imported states. For example, the symbols `pkg.mod.func1` and `pkg.mod.func2` may conflict when importing `pkg` or `pkg.mod`
```text
2026-04-07 16:35:30.108 ERROR [suite-thread-1] (ScriptContext.groovy:122) - Run test_pythonudtf_data_types_module in /mnt/disk7/linzhenqi/dv/doris/regression-test/suites/pythonudtf_p0/test_pythonudtf_data_types_module.groovy failed
java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Flight stream finish failed with message: 'pyudtf_module'. Detail: Python exception: Traceback (most recent call last):
File "pyarrow/_flight.pyx", line 2315, in pyarrow._flight._do_exchange
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2499, in do_exchange
self._handle_exchange_udtf(python_udf_meta, reader, writer)
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2316, in _handle_exchange_udtf
adaptive_udtf = loader.load()
^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 877, in load
func = self.load_udf_from_module(location, package_name, module_name, func_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1135, in load_udf_from_module
return self._load_package_udf(
^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1088, in _load_package_udf
udf_module = self._get_or_import_module(location, full_module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 978, in _get_or_import_module
module = importlib.import_module(full_module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/miniconda3/envs/py312/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 946, in _load_unlocked
KeyError: 'pyudtf_module'
...
2026-04-07 16:35:30.169 ERROR [suite-thread-6] (ScriptContext.groovy:122) - Run test_pythonudtf_exceptions_module in /mnt/disk7/linzhenqi/dv/doris/regression-test/suites/pythonudtf_p0/test_pythonudtf_exceptions_module.groovy failed
java.sql.SQLException: errCode = 2, detailMessage = (127.0.0.1)[RUNTIME_ERROR]Flight stream finish failed with message: 'pyudtf_module'. Detail: Python exception: Traceback (most recent call last):
File "pyarrow/_flight.pyx", line 2315, in pyarrow._flight._do_exchange
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2499, in do_exchange
self._handle_exchange_udtf(python_udf_meta, reader, writer)
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 2316, in _handle_exchange_udtf
adaptive_udtf = loader.load()
^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 877, in load
func = self.load_udf_from_module(location, package_name, module_name, func_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1135, in load_udf_from_module
return self._load_package_udf(
^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 1088, in _load_package_udf
udf_module = self._get_or_import_module(location, full_module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/dv/doris/output/be/plugins/python_udf/python_server.py", line 978, in _get_or_import_module
module = importlib.import_module(full_module_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/disk7/linzhenqi/miniconda3/envs/py312/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1337, in _find_and_load_unlocked
KeyError: 'pyudtf_module'
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4197943862
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4205034149
# BE Regression && UT Coverage Report
Increment line coverage `89.29% (25/28)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62167_0a14f1fd02917d5fdc21cf18357757d398c5e47e_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62167_0a14f1fd02917d5fdc21cf18357757d398c5e47e_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.68% (27382/37162) |
| Line Coverage | 57.30% (295434/515622) |
| Region Coverage | 54.37% (245319/451226) |
| Branch Coverage | 56.20% (106674/189815) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4203925487
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62167_0a14f1fd02917d5fdc21cf18357757d398c5e47e_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62167_0a14f1fd02917d5fdc21cf18357757d398c5e47e_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.68% (27382/37162) |
| Line Coverage | 57.29% (295420/515622) |
| Region Coverage | 54.34% (245215/451226) |
| Branch Coverage | 56.19% (106663/189815) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #62167:
URL: https://github.com/apache/doris/pull/62167#discussion_r3045326223
##########
be/src/udf/python/python_server.py:
##########
@@ -835,13 +835,13 @@ class ModuleUDFLoader(UDFLoader):
@classmethod
def _get_import_lock(cls, module_name: str) -> threading.Lock:
"""
- Get or create a reentrant lock for the given module name.
+ Get or create an import lock for the given module namespace.
Uses double-checked locking pattern for optimal performance:
- Fast path: return existing lock without acquiring global lock
- Slow path: create new lock under global lock protection
"""
- cache_key = module_name
+ cache_key = module_name.split(".", 1)[0]
Review Comment:
加个注释说明一下吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4205723923
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on code in PR #62167:
URL: https://github.com/apache/doris/pull/62167#discussion_r3048900817
##########
be/src/udf/python/python_server.py:
##########
@@ -835,13 +835,13 @@ class ModuleUDFLoader(UDFLoader):
@classmethod
def _get_import_lock(cls, module_name: str) -> threading.Lock:
"""
- Get or create a reentrant lock for the given module name.
+ Get or create an import lock for the given module namespace.
Uses double-checked locking pattern for optimal performance:
- Fast path: return existing lock without acquiring global lock
- Slow path: create new lock under global lock protection
"""
- cache_key = module_name
+ cache_key = module_name.split(".", 1)[0]
Review Comment:
fixed
##########
be/src/udf/python/python_server.py:
##########
@@ -835,13 +835,13 @@ class ModuleUDFLoader(UDFLoader):
@classmethod
def _get_import_lock(cls, module_name: str) -> threading.Lock:
"""
- Get or create a reentrant lock for the given module name.
+ Get or create an import lock for the given module namespace.
Uses double-checked locking pattern for optimal performance:
- Fast path: return existing lock without acquiring global lock
- Slow path: create new lock under global lock protection
"""
- cache_key = module_name
+ cache_key = module_name.split(".", 1)[0]
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4199431521
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62167_ee7b953fa5bead1a8a7d5deec6f04e2e33513836_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62167_ee7b953fa5bead1a8a7d5deec6f04e2e33513836_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.83% (26694/37162) |
| Line Coverage | 54.81% (282573/515549) |
| Region Coverage | 51.94% (234322/451146) |
| Branch Coverage | 53.34% (101232/189773) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb merged PR #62167:
URL: https://github.com/apache/doris/pull/62167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4198056783
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4197932022
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4203438150
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62167:
URL: https://github.com/apache/doris/pull/62167#discussion_r3044205471
##########
be/src/udf/python/python_server.py:
##########
@@ -835,13 +835,13 @@ class ModuleUDFLoader(UDFLoader):
@classmethod
def _get_import_lock(cls, module_name: str) -> threading.Lock:
"""
- Get or create a reentrant lock for the given module name.
+ Get or create an import lock for the given module namespace.
Uses double-checked locking pattern for optimal performance:
- Fast path: return existing lock without acquiring global lock
- Slow path: create new lock under global lock protection
"""
Review Comment:
This widens the import lock to the top-level package, which matches the fact that `_clear_modules_from_sys()` removes ancestor packages as well. The remaining gap is coverage: there is still no automated test that reproduces concurrent loads of `pkg.mod1` and `pkg.mod2` from separate requests, which is the exact `suite-thread-*` race described in the PR. `test_pythonudtf_pkg_isolation.groovy` is single-threaded, so this can regress silently. Please add a regression or python_server-level concurrency test for that failure mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4204432659
run nonConcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](pyudf) Fix import conflicts for modules with the same top-level name [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62167:
URL: https://github.com/apache/doris/pull/62167#issuecomment-4205723701
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org