You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2024/04/02 17:47:32 UTC
[I] [Failing Test]: JS2PY not supporting Python 3.12 [beam]
tvalentyn opened a new issue, #30829:
URL: https://github.com/apache/beam/issues/30829
### What happened?
Beam YAML added added a dependency on js2py library:
https://github.com/apache/beam/blob/eb5c73da86f87d65109ca84e917fdaaeb61f7cf1/sdks/python/apache_beam/yaml/yaml_mapping.py#L34
It currently doesn't work for Python 3.12:
- https://github.com/PiotrDabkowski/Js2Py/issues/282#issuecomment-1327954348
- https://github.com/PiotrDabkowski/Js2Py/issues/317
This means that YAML functionality might not be available in Python 3.12 until the issue is resolved. It seems that there has been no releases to Js2Py since 2022.
How critical is this dependency for Beam YAML? @Polber would you have time to investigate whether this dependency is still required and/or work with Js2Py maintainers on adding Python 3.12 support?
Thanks!
To repro, we need to make some patches to Beam repo like ones in #30828, create a py3.12 envrionment, run:
```
(py312) :python$ pytest apache_beam/yaml/programming_guide_test.py
=========================================================== test session starts ============================================================
platform linux -- Python 3.12.2, pytest-7.4.4, pluggy-1.4.0
rootdir: /home/valentyn/projects/beam/beam/beam/sdks/python
configfile: pytest.ini
plugins: xdist-3.5.0, requests-mock-1.12.1, timeout-2.3.1, hypothesis-6.100.0
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 0 items / 1 error
================================================================== ERRORS ==================================================================
_______________________________________ ERROR collecting apache_beam/yaml/programming_guide_test.py ________________________________________
/home/valentyn/.pyenv/versions/3.12.2/lib/python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
???
<frozen importlib._bootstrap>:1360: in _find_and_load
???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1387: in _gcd_import
???
<frozen importlib._bootstrap>:1360: in _find_and_load
???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:935: in _load_unlocked
???
<frozen importlib._bootstrap_external>:995: in exec_module
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
apache_beam/yaml/__init__.py:18: in <module>
from apache_beam.yaml.yaml_transform import *
apache_beam/yaml/yaml_transform.py:39: in <module>
from apache_beam.yaml.yaml_combine import normalize_combine
apache_beam/yaml/yaml_combine.py:33: in <module>
from apache_beam.yaml import yaml_mapping
apache_beam/yaml/yaml_mapping.py:34: in <module>
import js2py
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/__init__.py:72: in <module>
from .base import PyJsException
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/base.py:2965: in <module>
@Js
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/base.py:165: in Js
return PyJsFunction(val, FunctionPrototype)
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/base.py:1377: in __init__
cand = fix_js_args(func)
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:27: in fix_js_args
code = append_arguments(six.get_function_code(func), ('this', 'arguments'))
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:121: in append_arguments
arg = name_translations[inst.arg]
E KeyError: 3
============================================================= warnings summary =============================================================
apache_beam/__init__.py:74
/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/__init__.py:74: UserWarning: This version of Apache Beam has not been sufficiently tested on Python 3.12. You may encounter bugs or missing features.
warnings.warn(
<frozen importlib._bootstrap>:488
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
<frozen importlib._bootstrap>:488
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
../../../../../../.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:141
/home/valentyn/.pyenv/versions/3.12.2/envs/py312/lib/python3.12/site-packages/js2py/utils/injector.py:141: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
code_obj.co_firstlineno, code_obj.co_lnotab,
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= short test summary info ==========================================================
ERROR apache_beam/yaml/programming_guide_test.py - KeyError: 3
```
### Issue Failure
Failure: Test is continually failing
### Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: JS2PY not supporting Python 3.12 [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #30829: [Failing Test]: JS2PY not supporting Python 3.12
URL: https://github.com/apache/beam/issues/30829
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: JS2PY not supporting Python 3.12 [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #30829:
URL: https://github.com/apache/beam/issues/30829#issuecomment-2035173381
> I see no reason in disabling the feature for the time being
Ok. we could not install the dependency for certain Python versions by adding a version guard, and change the YAML functionality to support Javascript UDF if the necessary libraries are imported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: JS2PY not supporting Python 3.12 [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on issue #30829:
URL: https://github.com/apache/beam/issues/30829#issuecomment-2033335471
There are actually some known performance issues with that library that were recently discovered after running some load tests. I was planning on revisiting and possibly using a different library/solution for JavaScript UDF's. It was safeguarded by marking as "experimental" so I see no reason in disabling the feature for the time being.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: JS2PY not supporting Python 3.12 [beam]
Posted by "Polber (via GitHub)" <gi...@apache.org>.
Polber commented on issue #30829:
URL: https://github.com/apache/beam/issues/30829#issuecomment-2035656102
That makes sense to me. I added that logic in: https://github.com/apache/beam/pull/30843
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org