You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "bzablocki (via GitHub)" <gi...@apache.org> on 2023/04/26 15:25:51 UTC
[GitHub] [beam] bzablocki opened a new issue, #26433: [Bug]: Integration Tests fail on creating a Python 3.8 environment on Mac M1
bzablocki opened a new issue, #26433:
URL: https://github.com/apache/beam/issues/26433
### What happened?
Hi all. I'm working on Mac M1 and I noticed that when trying to run an arbitrary integration test using Python3.8 on Mac M1, I get the following error:
```
$ ./gradlew :sdks:python:test-suites:dataflow:py38:postCommitIT
>>> RUNNING integration tests with pipeline options: --runner=TestDataflowRunner --project=apache-beam-testing --region=us-central1 --staging_location=gs://temp-storage-for-end-to-end-tests/staging-it --temp_location=gs://temp-storage-for-end-to-end-tests/temp-it --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output --sdk_location=/Users/bzablocki/code/beam/sdks/python/build/apache-beam.tar.gz --requirements_file=postcommit_requirements.txt --num_workers=1 --sleep_secs=20 --kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test --dataflow_kms_key=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test
>>> pytest options: --capture=no --timeout=4500 --color=yes --log-cli-level=INFO --numprocesses=8 --dist=loadfile
>>> collect markers: -m=it_postcommit
thread '<unnamed>' panicked at 'Python API call failed', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.15.2/src/err/mod.rs:582:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
ImportError: dlopen(/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so, 0x0002): tried: '/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (no such file), '/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
Traceback (most recent call last):
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 189, in console_main
code = main()
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 147, in main
config = _prepareconfig(args, plugins)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 328, in _prepareconfig
config = pluginmanager.hook.pytest_cmdline_parse(
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
gen.send(outcome)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
config: Config = outcome.get_result()
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1067, in pytest_cmdline_parse
self.parse(args)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1354, in parse
self._preparse(args, addopts=addopts)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1256, in _preparse
self.hook.pytest_load_initial_conftests(
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
return outcome.get_result()
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
raise ex[1].with_traceback(ex[2])
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
res = hook_impl.function(*args)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1133, in pytest_load_initial_conftests
self.pluginmanager._set_initial_conftests(
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 559, in _set_initial_conftests
self._try_load_conftest(current, namespace.importmode, rootpath)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 573, in _try_load_conftest
self._getconftestmodules(anchor, importmode, rootpath)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 602, in _getconftestmodules
mod = self._importconftest(conftestpath, importmode, rootpath)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/config/__init__.py", line 634, in _importconftest
mod = import_path(conftestpath, mode=importmode, root=rootpath)
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/pathlib.py", line 564, in import_path
importlib.import_module(module_name)
File "/Users/bzablocki/.pyenv/versions/3.8.10/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 172, in exec_module
exec(co, module.__dict__)
File "/Users/bzablocki/code/beam/sdks/python/conftest.py", line 22, in <module>
from apache_beam.options import pipeline_options
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/__init__.py", line 88, in <module>
from apache_beam import io
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/__init__.py", line 21, in <module>
from apache_beam.io.avroio import *
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/avroio.py", line 53, in <module>
from apache_beam.io import filebasedsink
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/filebasedsink.py", line 29, in <module>
from apache_beam.io import iobase
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/iobase.py", line 54, in <module>
from apache_beam.transforms import Impulse
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/transforms/__init__.py", line 24, in <module>
from apache_beam.transforms.external import *
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/transforms/external.py", line 44, in <module>
from apache_beam.runners import pipeline_context
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/runners/__init__.py", line 29, in <module>
from apache_beam.runners.dataflow.dataflow_runner import DataflowRunner
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/runners/dataflow/__init__.py", line 24, in <module>
from apache_beam.runners.dataflow.dataflow_runner import DataflowRunner
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 87, in <module>
class DataflowRunner(PipelineRunner):
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 106, in DataflowRunner
from apache_beam.runners.dataflow.ptransform_overrides import CombineValuesPTransformOverride
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/runners/dataflow/ptransform_overrides.py", line 23, in <module>
from apache_beam.pipeline import PTransformOverride
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/pipeline.py", line 76, in <module>
from apache_beam.io.filesystems import FileSystems
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/filesystems.py", line 47, in <module>
from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/gcp/gcsfilesystem.py", line 36, in <module>
from apache_beam.io.gcp import gcsio
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/io/gcp/gcsio.py", line 70, in <module>
from apache_beam.internal.gcp import auth
File "/Users/bzablocki/code/beam/sdks/python/apache_beam/internal/gcp/auth.py", line 32, in <module>
from google.auth import impersonated_credentials
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/google/auth/impersonated_credentials.py", line 39, in <module>
from google.auth import jwt
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/google/auth/jwt.py", line 57, in <module>
from google.auth import _service_account_info
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/google/auth/_service_account_info.py", line 22, in <module>
from google.auth import crypt
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/google/auth/crypt/__init__.py", line 43, in <module>
from google.auth.crypt import rsa
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/google/auth/crypt/rsa.py", line 20, in <module>
from google.auth.crypt import _cryptography_rsa
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/google/auth/crypt/_cryptography_rsa.py", line 25, in <module>
from cryptography.hazmat.primitives import serialization
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/cryptography/hazmat/primitives/serialization/__init__.py", line 16, in <module>
from cryptography.hazmat.primitives.serialization.base import (
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 9, in <module>
from cryptography.hazmat.primitives.asymmetric.types import (
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/types.py", line 8, in <module>
from cryptography.hazmat.primitives.asymmetric import (
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py", line 10, in <module>
from cryptography.hazmat.primitives.asymmetric import utils as asym_utils
File "/Users/bzablocki/code/beam/build/gradleenv/-1734967051/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/utils.py", line 6, in <module>
from cryptography.hazmat.bindings._rust import asn1
pyo3_runtime.PanicException: Python API call failed
```
Running the same integration test (and the associated gradle task) in Python 3.9 and Python 3.10 works fine.
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn closed issue #26433: [Bug]: Integration Tests fail on creating a Python 3.8 environment on Mac M1
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #26433: [Bug]: Integration Tests fail on creating a Python 3.8 environment on Mac M1
URL: https://github.com/apache/beam/issues/26433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #26433: [Bug]: Integration Tests fail on creating a Python 3.8 environment on Mac M1
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26433:
URL: https://github.com/apache/beam/issues/26433#issuecomment-1540346403
You could try to dig further and/or reach out to `cryptography` maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #26433: [Bug]: Integration Tests fail on creating a Python 3.8 environment on Mac M1
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26433:
URL: https://github.com/apache/beam/issues/26433#issuecomment-1540344645
Thanks for reporting. We've hit many issues with M1's in the past but haven't seen this one. Unfortunately this looks unactionable for Beam, especially since it doesn't fail on newer versions of Python. You might be able to fix it by installing some extra dependencies, but not sure which ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org