You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Valentyn Tymofieiev (Jira)" <ji...@apache.org> on 2019/10/31 17:57:00 UTC
[jira] [Resolved] (BEAM-8397)
DataflowRunnerTest.test_remote_runner_display_data sometimes fails due to
infinite recursion during pickling on Python 3.7.
[ https://issues.apache.org/jira/browse/BEAM-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Valentyn Tymofieiev resolved BEAM-8397.
---------------------------------------
Fix Version/s: 2.17.0
Resolution: Fixed
> DataflowRunnerTest.test_remote_runner_display_data sometimes fails due to infinite recursion during pickling on Python 3.7.
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-8397
> URL: https://issues.apache.org/jira/browse/BEAM-8397
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Fix For: 2.17.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> `python ./setup.py test -s apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data` passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam depends on dill==0.3.1.1.`python ./setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data` fails currently if run on master.
> The failure indicates infinite recursion during pickling:
> {noformat}
> test_remote_runner_display_data (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot recover from stack overflow.
> Current thread 0x00007f9d700ed740 (most recent call first):
> File "/usr/lib/python3.7/pickle.py", line 479 in get
> File "/usr/lib/python3.7/pickle.py", line 497 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 910 in save_module_dict
> File "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py", line 198 in new_save_module_dict
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py", line 114 in wrapper
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 1137 in save_cell
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 771 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 786 in save_tuple
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 638 in save_reduce
> File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 1394 in save_function
> File "/usr/lib/python3.7/pickle.py", line 504 in save
> File "/usr/lib/python3.7/pickle.py", line 882 in _batch_setitems
> File "/usr/lib/python3.7/pickle.py", line 856 in save_dict
> File "/usr/local/google/home/valentyn/tmp/py37env/lib/python3.7/site-packages/dill/_dill.py", line 910 in save_module_dict
> File "/usr/local/google/home/valentyn/projects/beam/clean/beam/sdks/python/apache_beam/internal/pickler.py", line 198 in new_save_module_dict
> ...
> {noformat}
> cc: [~yoshiki.obata]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)