Re: [I] [Bug]: Python BQ directrunner postcommits flake with KeyError: '__pyx_vtable__' [beam]

["tvalentyn (via GitHub)" <gi...@apache.org>]

tvalentyn commented on issue #29617:
URL: https://github.com/apache/beam/issues/29617#issuecomment-2134229550

   > understand what started to trigger it.
   
   One possibility is that we started to use wheels in our integration tests, and perhaps the built wheel is not compatible with GHA platform, or it is not built reliably, or somehow broken.
   
   To investigate further we could:
   
   1) Create a PR that limits the postcommit suite to only direct runner tests, it should be a modification to gradle files in https://github.com/apache/beam/blob/e764fc9c17dbc575d4a16b5e6454f4d38ef752be/build.gradle.kts#L506 , and this modification will be picked up by GHA runner when a non-committer modifies the trigger file: https://github.com/apache/beam/blob/master/.github/trigger_files/beam_PostCommit_Python.json 
   
   2) It would help to print or inspect `sha` of the created wheel, and see if it stays consistent between passing and failing runs of this test, when there are no other changes to the code.
   
   3) We could see if error disappears if we use sources instead of wheels. For example we can avoid this branch that builds the wheel: https://github.com/apache/beam/blob/e764fc9c17dbc575d4a16b5e6454f4d38ef752be/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L3066
   
   note that logs below also say something about "reparing" the wheel, although it might be a red herring.
   
   ```
   INFO:auditwheel.main_repair:Repairing apache_beam-2.57.0.dev0-cp39-cp39-linux_x86_64.whl
   #9 156.8   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/runners/worker/operations.pxd
   #9 156.8     tree = Parsing.p_module(s, pxd, full_module_name)
   INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
   INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_x86_64, manylinux2014_x86_64
   INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_x86_64
   INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux_2_17_x86_64, cp39-cp39-manylinux2014_x86_64
   #9 158.4   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/runners/worker/statesampler_fast.pxd
   #9 158.4     tree = Parsing.p_module(s, pxd, full_module_name)
   #9 159.0   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/testing/fast_test_utils.pxd
   #9 159.0     tree = Parsing.p_module(s, pxd, full_module_name)
   #9 159.1   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/transforms/cy_combiners.pxd
   #9 159.1     tree = Parsing.p_module(s, pxd, full_module_name)
   #9 159.7   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/transforms/cy_dataflow_distribution_counter.pxd
   #9 159.7     tree = Parsing.p_module(s, pxd, full_module_name)
   #9 159.8   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/transforms/stats.pxd
   #9 159.8     tree = Parsing.p_module(s, pxd, full_module_name)
   #9 160.1   /tmp/pip-build-env-x6bammmu/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-req-build-i5vxycdj/apache_beam/utils/counters.pxd
   #9 160.1     tree = Parsing.p_module(s, pxd, full_module_name)
   INFO:auditwheel.main_repair:
   Fixed-up wheel written to /tmp/cibuildwheel/repaired_wheel/apache_beam-2.57.0.dev0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org