You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ning Kang <ni...@google.com> on 2019/10/11 23:55:00 UTC
Fatal Python error: Cannot recover from stack overflow
Hi everyone,
I'm blocked by a stack overflow error from Python test:
test_remote_runner_display_data
(apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
And I've also toyed with *tox.ini* to run the test with a clean beam repo.
To run the tests:
*gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
To speed up the process by only running the test with the issue, you can
change the tox.ini from line 153:
[testenv:py37-gcp]
setenv =
RUN_SKIPPED_PY3_TESTS=0
extras = test,gcp
commands =
# python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
python setup.py nosetests --tests
'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'
{posargs}
It turns out this stack overflow error occurs even in a clean beam repo if
you just run the single test.
However, if you just run the original py37-gcp test (the commented line),
everything passes.
In that scenario, by checking the generated *nosetests.xml*, we can see the
test throwing error is skipped: <skipped type="unittest.case.SkipTest"
message="GCP dependencies are not installed"><![CDATA[Exception: GCP
dependencies are not installed
However, py37-gcp does have gcp dependency and should not skip the test.
I'm wondering if the error exists long time ago. My change has nothing to
do with dataflow runner (in completely different packages and new files are
not referenced at all). But somehow, the change triggers the test to be
run. Similar to modifying the *tox.ini* file, that also trigger the test to
be run.
Attached stack trace. Looks like _dill.py and pickle.py are calling each
other into an infinite loop.
test_remote_runner_display_data (apache_beam.runners.dataflow.
dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot
recover from stack overflow.
Current thread 0x0000000112f645c0 (most recent call first):
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 479 in get
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 497 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1394 in save_function
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 885 in _batch_setitems
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 859 in save_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 910 in save_module_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 198 in new_save_module_dict
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 114 in wrapper
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1137 in save_cell
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1394 in save_function
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 885 in _batch_setitems
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 859 in save_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 910 in save_module_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 198 in new_save_module_dict
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 114 in wrapper
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1137 in save_cell
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1394 in save_function
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 885 in _batch_setitems
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 859 in save_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 910 in save_module_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 198 in new_save_module_dict
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 114 in wrapper
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1137 in save_cell
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1394 in save_function
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 885 in _batch_setitems
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 859 in save_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 910 in save_module_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 198 in new_save_module_dict
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 114 in wrapper
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1137 in save_cell
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 774 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 789 in save_tuple
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 638 in save_reduce
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 1394 in save_function
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 885 in _batch_setitems
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 859 in save_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
, line 910 in save_module_dict
File
"/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
, line 198 in new_save_module_dict
File
"/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
, line 504 in save
...
ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/python/test-
suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/bin/python
setup.py nosetests --tests apache_beam/runners/dataflow/dataflow_runner_test
.py:DataflowRunnerTest.test_remote_runner_display_data (exited with code -6)
py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/python/
test-suites/tox/py37/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Ning Kang <ni...@google.com>.
Thank you very much for the investigation, Valentyn!
Yes, I can change the ipython version. And it works!
On Mon, Oct 14, 2019 at 5:18 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> By reproducing locally in last email, I meant that I can reproduce it in
> tox as well. I iterated on tox runs via:,
>
> rm -rf ./target
> tox -e py37-gcp
>
> I further narrowed down that adding a dependency on ipython triggers this
> error.
>
> A workaround that may get you unblocked in short term is to set
> 'ipython>=5.8.0,<6'. Would that work for you in the meantime?
>
> If that works, we can try to further narrow down the ipython version that
> triggers the error, and continue to investigate what causes this error.
>
> I suspect some part of the codebase is brittle and we need to fix it.
>
>
> On Mon, Oct 14, 2019 at 4:01 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks, I can reproduce your error locally but did not find the
>> rootcause yet, I continue looking into this issue.
>>
>> On Mon, Oct 14, 2019 at 1:04 PM Ning Kang <ni...@google.com> wrote:
>>
>>> Thanks Pablo and Valentyn for the help and investigation!
>>>
>>> My gradle scan:
>>> https://scans.gradle.com/s/i42lzbzy44hio/console-log?task=:sdks:python:test-suites:tox:py37:testPy37Gcp
>>> .
>>> Somehow, the error is acting up for my pre-commit check of
>>> https://github.com/apache/beam/pull/9741.
>>>
>>>
>>> On Mon, Oct 14, 2019 at 12:24 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>>
>>>> Thanks, I can also reproduce this error.
>>>>
>>>> It appears that running this test through nose makes the test fail on
>>>> Python 3.7, however the test does not fail on its own, and does not fail in
>>>> a tox suite. However the test starts failing in tox if we upgrade to
>>>> dill==0.3.1.1, so I associated this failure with an upgrade to
>>>> dill==0.3.1.1 in the past. To summarize:
>>>>
>>>> `python ./setup.py test -s
>>>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>>>> passes.
>>>> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if
>>>> Beam depends on dill==0.3.1.1.
>>>> `python ./setup.py nosetests --tests
>>>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
>>>> fails currently if run on master.
>>>>
>>>> I filed https://issues.apache.org/jira/browse/BEAM-8397 to investigate
>>>> this. You mentioned that you are currently blocked on this error - are you
>>>> not able to submit a PR or in what way are you blocked? My understanding is
>>>> that this error is not surfacing in precommit tests yet.
>>>>
>>>> Thanks,
>>>> Valentyn
>>>>
>>>>
>>>> On Mon, Oct 14, 2019 at 11:48 AM Ning Kang <ni...@google.com> wrote:
>>>>
>>>>> I tried installing with `pip install -e .[gcp,test]` locally from a
>>>>> cloned repo without my changes. Still no luck.
>>>>> And when running the task, there would be `setupVirtualenv` task to
>>>>> setup the dependencies, so the gcp dependency should be available.
>>>>> I don't think the dependency is the problem because otherwise, the
>>>>> test will be skipped and I won't run into the error.
>>>>> I've checked the `dill` installed is 0.3.0. Force re-installed
>>>>> dependencies, force rerun the tasks.
>>>>>
>>>>> The only time the test passes with gradle and tox for me is when I run
>>>>> the task `*:sdks:python:test-suites:tox:py37:testPy37Gcp*` including
>>>>> all the tests with `python setup.py nosetests --ignore-files
>>>>> '.*py3[8-9]\.py$' {posargs}` from a clean repo.
>>>>> Also, cmd `python ./setup.py test -s
>>>>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>>>>> shows OK for me with and without my change.
>>>>> But if I use nosetests, cmd `python ./setup.py nosetests --tests
>>>>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'`
>>>>> will always stack overflow for either with or without my change.
>>>>>
>>>>> The issue looks like the stack overflow problem Valentyn has commented
>>>>> on this issue: https://issues.apache.org/jira/browse/BEAM-8324
>>>>>
>>>>> I can't really tell what's happening behind the scenes. Like how a
>>>>> test passes and fails when invoked with different tools? (Note: the test
>>>>> will be skipped if dependencies are not setup)
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2019 at 5:51 PM Pablo Estrada <pa...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Silly question, but how did you install Beam? did you use pip install
>>>>>> -e .[gcp,test]?
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
>>>>>>
>>>>>>> Thanks Valentyn, I've tried rebasing several times and caught up
>>>>>>> with upstream several times from yesterday but the issue is not gone.
>>>>>>> I can see the dill is pinned to 0.3.0. And this is true in my code
>>>>>>> base and in clean beam repo (freshly cloned today).
>>>>>>> I just don't know how the suite skipped/passed the tests at current
>>>>>>> master. Because when the test is run, there is the failure.
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>>> Try to rebase your branch on top of master (or just change this
>>>>>>>> line [2] )
>>>>>>>>
>>>>>>>> There is an issue with dill==0.3.1.1 that we did not rootcause yet
>>>>>>>> [1], but we have pinned dill to 0.3.0 on master for now.
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-5878
>>>>>>>>
>>>>>>>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>>>>>>>
>>>>>>>>> Hi everyone,
>>>>>>>>>
>>>>>>>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>>>>>>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>>>>>>>
>>>>>>>>> And I've also toyed with *tox.ini* to run the test with a clean
>>>>>>>>> beam repo.
>>>>>>>>> To run the tests:
>>>>>>>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>>>>>>>> To speed up the process by only running the test with the issue,
>>>>>>>>> you can change the tox.ini from line 153:
>>>>>>>>>
>>>>>>>>> [testenv:py37-gcp]
>>>>>>>>> setenv =
>>>>>>>>> RUN_SKIPPED_PY3_TESTS=0
>>>>>>>>> extras = test,gcp
>>>>>>>>> commands =
>>>>>>>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>>>>>>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>>>>>>>
>>>>>>>>> It turns out this stack overflow error occurs even in a clean beam
>>>>>>>>> repo if you just run the single test.
>>>>>>>>> However, if you just run the original py37-gcp test (the commented
>>>>>>>>> line), everything passes.
>>>>>>>>> In that scenario, by checking the generated *nosetests.xml*, we
>>>>>>>>> can see the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>>>>>>>> message="GCP dependencies are not installed"><![CDATA[Exception:
>>>>>>>>> GCP dependencies are not installed
>>>>>>>>> However, py37-gcp does have gcp dependency and should not skip the
>>>>>>>>> test.
>>>>>>>>>
>>>>>>>>> I'm wondering if the error exists long time ago. My change has
>>>>>>>>> nothing to do with dataflow runner (in completely different packages and
>>>>>>>>> new files are not referenced at all). But somehow, the change triggers the
>>>>>>>>> test to be run. Similar to modifying the *tox.ini* file, that
>>>>>>>>> also trigger the test to be run.
>>>>>>>>>
>>>>>>>>> Attached stack trace. Looks like _dill.py and pickle.py are
>>>>>>>>> calling each other into an infinite loop.
>>>>>>>>>
>>>>>>>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>>>>>>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error:
>>>>>>>>> Cannot recover from stack overflow.
>>>>>>>>>
>>>>>>>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 479 in get
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 497 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1394 in save_function
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 859 in save_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 910 in save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 114 in wrapper
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1137 in save_cell
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1394 in save_function
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 859 in save_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 910 in save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 114 in wrapper
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1137 in save_cell
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1394 in save_function
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 859 in save_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 910 in save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 114 in wrapper
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1137 in save_cell
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1394 in save_function
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 859 in save_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 910 in save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 114 in wrapper
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1137 in save_cell
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 774 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 789 in save_tuple
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 638 in save_reduce
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 1394 in save_function
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 859 in save_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>>> , line 910 in save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>>
>>>>>>>>> File
>>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>>> , line 504 in save
>>>>>>>>>
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/
>>>>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-
>>>>>>>>> py37-gcp/py37-gcp/bin/python setup.py nosetests --tests
>>>>>>>>> apache_beam/runners/dataflow/dataflow_runner_test.py:
>>>>>>>>> DataflowRunnerTest.test_remote_runner_display_data (exited with
>>>>>>>>> code -6)
>>>>>>>>>
>>>>>>>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks
>>>>>>>>> /python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>>>>>>>> run_tox_cleanup.sh
>>>>>>>>>
>>>>>>>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Valentyn Tymofieiev <va...@google.com>.
By reproducing locally in last email, I meant that I can reproduce it in
tox as well. I iterated on tox runs via:,
rm -rf ./target
tox -e py37-gcp
I further narrowed down that adding a dependency on ipython triggers this
error.
A workaround that may get you unblocked in short term is to set
'ipython>=5.8.0,<6'. Would that work for you in the meantime?
If that works, we can try to further narrow down the ipython version that
triggers the error, and continue to investigate what causes this error.
I suspect some part of the codebase is brittle and we need to fix it.
On Mon, Oct 14, 2019 at 4:01 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thanks, I can reproduce your error locally but did not find the rootcause
> yet, I continue looking into this issue.
>
> On Mon, Oct 14, 2019 at 1:04 PM Ning Kang <ni...@google.com> wrote:
>
>> Thanks Pablo and Valentyn for the help and investigation!
>>
>> My gradle scan:
>> https://scans.gradle.com/s/i42lzbzy44hio/console-log?task=:sdks:python:test-suites:tox:py37:testPy37Gcp
>> .
>> Somehow, the error is acting up for my pre-commit check of
>> https://github.com/apache/beam/pull/9741.
>>
>>
>> On Mon, Oct 14, 2019 at 12:24 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Thanks, I can also reproduce this error.
>>>
>>> It appears that running this test through nose makes the test fail on
>>> Python 3.7, however the test does not fail on its own, and does not fail in
>>> a tox suite. However the test starts failing in tox if we upgrade to
>>> dill==0.3.1.1, so I associated this failure with an upgrade to
>>> dill==0.3.1.1 in the past. To summarize:
>>>
>>> `python ./setup.py test -s
>>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>>> passes.
>>> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if
>>> Beam depends on dill==0.3.1.1.
>>> `python ./setup.py nosetests --tests
>>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
>>> fails currently if run on master.
>>>
>>> I filed https://issues.apache.org/jira/browse/BEAM-8397 to investigate
>>> this. You mentioned that you are currently blocked on this error - are you
>>> not able to submit a PR or in what way are you blocked? My understanding is
>>> that this error is not surfacing in precommit tests yet.
>>>
>>> Thanks,
>>> Valentyn
>>>
>>>
>>> On Mon, Oct 14, 2019 at 11:48 AM Ning Kang <ni...@google.com> wrote:
>>>
>>>> I tried installing with `pip install -e .[gcp,test]` locally from a
>>>> cloned repo without my changes. Still no luck.
>>>> And when running the task, there would be `setupVirtualenv` task to
>>>> setup the dependencies, so the gcp dependency should be available.
>>>> I don't think the dependency is the problem because otherwise, the test
>>>> will be skipped and I won't run into the error.
>>>> I've checked the `dill` installed is 0.3.0. Force re-installed
>>>> dependencies, force rerun the tasks.
>>>>
>>>> The only time the test passes with gradle and tox for me is when I run
>>>> the task `*:sdks:python:test-suites:tox:py37:testPy37Gcp*` including
>>>> all the tests with `python setup.py nosetests --ignore-files
>>>> '.*py3[8-9]\.py$' {posargs}` from a clean repo.
>>>> Also, cmd `python ./setup.py test -s
>>>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>>>> shows OK for me with and without my change.
>>>> But if I use nosetests, cmd `python ./setup.py nosetests --tests
>>>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'`
>>>> will always stack overflow for either with or without my change.
>>>>
>>>> The issue looks like the stack overflow problem Valentyn has commented
>>>> on this issue: https://issues.apache.org/jira/browse/BEAM-8324
>>>>
>>>> I can't really tell what's happening behind the scenes. Like how a test
>>>> passes and fails when invoked with different tools? (Note: the test will be
>>>> skipped if dependencies are not setup)
>>>>
>>>>
>>>> On Fri, Oct 11, 2019 at 5:51 PM Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>>
>>>>> Silly question, but how did you install Beam? did you use pip install
>>>>> -e .[gcp,test]?
>>>>>
>>>>> On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
>>>>>
>>>>>> Thanks Valentyn, I've tried rebasing several times and caught up with
>>>>>> upstream several times from yesterday but the issue is not gone.
>>>>>> I can see the dill is pinned to 0.3.0. And this is true in my code
>>>>>> base and in clean beam repo (freshly cloned today).
>>>>>> I just don't know how the suite skipped/passed the tests at current
>>>>>> master. Because when the test is run, there is the failure.
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>>> Try to rebase your branch on top of master (or just change this line
>>>>>>> [2] )
>>>>>>>
>>>>>>> There is an issue with dill==0.3.1.1 that we did not rootcause yet
>>>>>>> [1], but we have pinned dill to 0.3.0 on master for now.
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/BEAM-5878
>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>>>>>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>>>>>>
>>>>>>>> And I've also toyed with *tox.ini* to run the test with a clean
>>>>>>>> beam repo.
>>>>>>>> To run the tests:
>>>>>>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>>>>>>> To speed up the process by only running the test with the issue,
>>>>>>>> you can change the tox.ini from line 153:
>>>>>>>>
>>>>>>>> [testenv:py37-gcp]
>>>>>>>> setenv =
>>>>>>>> RUN_SKIPPED_PY3_TESTS=0
>>>>>>>> extras = test,gcp
>>>>>>>> commands =
>>>>>>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>>>>>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>>>>>>
>>>>>>>> It turns out this stack overflow error occurs even in a clean beam
>>>>>>>> repo if you just run the single test.
>>>>>>>> However, if you just run the original py37-gcp test (the commented
>>>>>>>> line), everything passes.
>>>>>>>> In that scenario, by checking the generated *nosetests.xml*, we
>>>>>>>> can see the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>>>>>>> message="GCP dependencies are not installed"><![CDATA[Exception:
>>>>>>>> GCP dependencies are not installed
>>>>>>>> However, py37-gcp does have gcp dependency and should not skip the
>>>>>>>> test.
>>>>>>>>
>>>>>>>> I'm wondering if the error exists long time ago. My change has
>>>>>>>> nothing to do with dataflow runner (in completely different packages and
>>>>>>>> new files are not referenced at all). But somehow, the change triggers the
>>>>>>>> test to be run. Similar to modifying the *tox.ini* file, that also
>>>>>>>> trigger the test to be run.
>>>>>>>>
>>>>>>>> Attached stack trace. Looks like _dill.py and pickle.py are calling
>>>>>>>> each other into an infinite loop.
>>>>>>>>
>>>>>>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>>>>>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error:
>>>>>>>> Cannot recover from stack overflow.
>>>>>>>>
>>>>>>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 479 in get
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 497 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1394 in save_function
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 859 in save_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 910 in save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 114 in wrapper
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1137 in save_cell
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1394 in save_function
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 859 in save_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 910 in save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 114 in wrapper
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1137 in save_cell
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1394 in save_function
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 859 in save_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 910 in save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 114 in wrapper
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1137 in save_cell
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1394 in save_function
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 859 in save_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 910 in save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 114 in wrapper
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1137 in save_cell
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 774 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 789 in save_tuple
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 638 in save_reduce
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 1394 in save_function
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 885 in _batch_setitems
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 859 in save_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>>> , line 910 in save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>>> , line 198 in new_save_module_dict
>>>>>>>>
>>>>>>>> File
>>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>>> , line 504 in save
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/
>>>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37
>>>>>>>> -gcp/py37-gcp/bin/python setup.py nosetests --tests apache_beam/
>>>>>>>> runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>>>>>>>> (exited with code -6)
>>>>>>>>
>>>>>>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/
>>>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>>>>>>> run_tox_cleanup.sh
>>>>>>>>
>>>>>>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Valentyn Tymofieiev <va...@google.com>.
Thanks, I can reproduce your error locally but did not find the rootcause
yet, I continue looking into this issue.
On Mon, Oct 14, 2019 at 1:04 PM Ning Kang <ni...@google.com> wrote:
> Thanks Pablo and Valentyn for the help and investigation!
>
> My gradle scan:
> https://scans.gradle.com/s/i42lzbzy44hio/console-log?task=:sdks:python:test-suites:tox:py37:testPy37Gcp
> .
> Somehow, the error is acting up for my pre-commit check of
> https://github.com/apache/beam/pull/9741.
>
>
> On Mon, Oct 14, 2019 at 12:24 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Thanks, I can also reproduce this error.
>>
>> It appears that running this test through nose makes the test fail on
>> Python 3.7, however the test does not fail on its own, and does not fail in
>> a tox suite. However the test starts failing in tox if we upgrade to
>> dill==0.3.1.1, so I associated this failure with an upgrade to
>> dill==0.3.1.1 in the past. To summarize:
>>
>> `python ./setup.py test -s
>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>> passes.
>> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if
>> Beam depends on dill==0.3.1.1.
>> `python ./setup.py nosetests --tests
>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
>> fails currently if run on master.
>>
>> I filed https://issues.apache.org/jira/browse/BEAM-8397 to investigate
>> this. You mentioned that you are currently blocked on this error - are you
>> not able to submit a PR or in what way are you blocked? My understanding is
>> that this error is not surfacing in precommit tests yet.
>>
>> Thanks,
>> Valentyn
>>
>>
>> On Mon, Oct 14, 2019 at 11:48 AM Ning Kang <ni...@google.com> wrote:
>>
>>> I tried installing with `pip install -e .[gcp,test]` locally from a
>>> cloned repo without my changes. Still no luck.
>>> And when running the task, there would be `setupVirtualenv` task to
>>> setup the dependencies, so the gcp dependency should be available.
>>> I don't think the dependency is the problem because otherwise, the test
>>> will be skipped and I won't run into the error.
>>> I've checked the `dill` installed is 0.3.0. Force re-installed
>>> dependencies, force rerun the tasks.
>>>
>>> The only time the test passes with gradle and tox for me is when I run
>>> the task `*:sdks:python:test-suites:tox:py37:testPy37Gcp*` including
>>> all the tests with `python setup.py nosetests --ignore-files
>>> '.*py3[8-9]\.py$' {posargs}` from a clean repo.
>>> Also, cmd `python ./setup.py test -s
>>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>>> shows OK for me with and without my change.
>>> But if I use nosetests, cmd `python ./setup.py nosetests --tests
>>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'`
>>> will always stack overflow for either with or without my change.
>>>
>>> The issue looks like the stack overflow problem Valentyn has commented
>>> on this issue: https://issues.apache.org/jira/browse/BEAM-8324
>>>
>>> I can't really tell what's happening behind the scenes. Like how a test
>>> passes and fails when invoked with different tools? (Note: the test will be
>>> skipped if dependencies are not setup)
>>>
>>>
>>> On Fri, Oct 11, 2019 at 5:51 PM Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Silly question, but how did you install Beam? did you use pip install
>>>> -e .[gcp,test]?
>>>>
>>>> On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
>>>>
>>>>> Thanks Valentyn, I've tried rebasing several times and caught up with
>>>>> upstream several times from yesterday but the issue is not gone.
>>>>> I can see the dill is pinned to 0.3.0. And this is true in my code
>>>>> base and in clean beam repo (freshly cloned today).
>>>>> I just don't know how the suite skipped/passed the tests at current
>>>>> master. Because when the test is run, there is the failure.
>>>>>
>>>>> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>>> Try to rebase your branch on top of master (or just change this line
>>>>>> [2] )
>>>>>>
>>>>>> There is an issue with dill==0.3.1.1 that we did not rootcause yet
>>>>>> [1], but we have pinned dill to 0.3.0 on master for now.
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/BEAM-5878
>>>>>>
>>>>>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>>>>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>>>>>
>>>>>>> And I've also toyed with *tox.ini* to run the test with a clean
>>>>>>> beam repo.
>>>>>>> To run the tests:
>>>>>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>>>>>> To speed up the process by only running the test with the issue, you
>>>>>>> can change the tox.ini from line 153:
>>>>>>>
>>>>>>> [testenv:py37-gcp]
>>>>>>> setenv =
>>>>>>> RUN_SKIPPED_PY3_TESTS=0
>>>>>>> extras = test,gcp
>>>>>>> commands =
>>>>>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>>>>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>>>>>
>>>>>>> It turns out this stack overflow error occurs even in a clean beam
>>>>>>> repo if you just run the single test.
>>>>>>> However, if you just run the original py37-gcp test (the commented
>>>>>>> line), everything passes.
>>>>>>> In that scenario, by checking the generated *nosetests.xml*, we can
>>>>>>> see the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>>>>>> message="GCP dependencies are not installed"><![CDATA[Exception:
>>>>>>> GCP dependencies are not installed
>>>>>>> However, py37-gcp does have gcp dependency and should not skip the
>>>>>>> test.
>>>>>>>
>>>>>>> I'm wondering if the error exists long time ago. My change has
>>>>>>> nothing to do with dataflow runner (in completely different packages and
>>>>>>> new files are not referenced at all). But somehow, the change triggers the
>>>>>>> test to be run. Similar to modifying the *tox.ini* file, that also
>>>>>>> trigger the test to be run.
>>>>>>>
>>>>>>> Attached stack trace. Looks like _dill.py and pickle.py are calling
>>>>>>> each other into an infinite loop.
>>>>>>>
>>>>>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>>>>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error:
>>>>>>> Cannot recover from stack overflow.
>>>>>>>
>>>>>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 479 in get
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 497 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1394 in save_function
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 885 in _batch_setitems
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 859 in save_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 910 in save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 198 in new_save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 114 in wrapper
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1137 in save_cell
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1394 in save_function
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 885 in _batch_setitems
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 859 in save_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 910 in save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 198 in new_save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 114 in wrapper
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1137 in save_cell
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1394 in save_function
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 885 in _batch_setitems
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 859 in save_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 910 in save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 198 in new_save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 114 in wrapper
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1137 in save_cell
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1394 in save_function
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 885 in _batch_setitems
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 859 in save_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 910 in save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 198 in new_save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 114 in wrapper
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1137 in save_cell
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 774 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 789 in save_tuple
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 638 in save_reduce
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 1394 in save_function
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 885 in _batch_setitems
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 859 in save_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>>> , line 910 in save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>>> , line 198 in new_save_module_dict
>>>>>>>
>>>>>>> File
>>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>>> , line 504 in save
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/
>>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-
>>>>>>> gcp/py37-gcp/bin/python setup.py nosetests --tests apache_beam/
>>>>>>> runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>>>>>>> (exited with code -6)
>>>>>>>
>>>>>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/
>>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>>>>>> run_tox_cleanup.sh
>>>>>>>
>>>>>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Ning Kang <ni...@google.com>.
Thanks Pablo and Valentyn for the help and investigation!
My gradle scan:
https://scans.gradle.com/s/i42lzbzy44hio/console-log?task=:sdks:python:test-suites:tox:py37:testPy37Gcp
.
Somehow, the error is acting up for my pre-commit check of
https://github.com/apache/beam/pull/9741.
On Mon, Oct 14, 2019 at 12:24 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Thanks, I can also reproduce this error.
>
> It appears that running this test through nose makes the test fail on
> Python 3.7, however the test does not fail on its own, and does not fail in
> a tox suite. However the test starts failing in tox if we upgrade to
> dill==0.3.1.1, so I associated this failure with an upgrade to
> dill==0.3.1.1 in the past. To summarize:
>
> `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> passes.
> `tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
> depends on dill==0.3.1.1.
> `python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
> fails currently if run on master.
>
> I filed https://issues.apache.org/jira/browse/BEAM-8397 to investigate
> this. You mentioned that you are currently blocked on this error - are you
> not able to submit a PR or in what way are you blocked? My understanding is
> that this error is not surfacing in precommit tests yet.
>
> Thanks,
> Valentyn
>
>
> On Mon, Oct 14, 2019 at 11:48 AM Ning Kang <ni...@google.com> wrote:
>
>> I tried installing with `pip install -e .[gcp,test]` locally from a
>> cloned repo without my changes. Still no luck.
>> And when running the task, there would be `setupVirtualenv` task to setup
>> the dependencies, so the gcp dependency should be available.
>> I don't think the dependency is the problem because otherwise, the test
>> will be skipped and I won't run into the error.
>> I've checked the `dill` installed is 0.3.0. Force re-installed
>> dependencies, force rerun the tasks.
>>
>> The only time the test passes with gradle and tox for me is when I run
>> the task `*:sdks:python:test-suites:tox:py37:testPy37Gcp*` including all
>> the tests with `python setup.py nosetests --ignore-files
>> '.*py3[8-9]\.py$' {posargs}` from a clean repo.
>> Also, cmd `python ./setup.py test -s
>> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
>> shows OK for me with and without my change.
>> But if I use nosetests, cmd `python ./setup.py nosetests --tests
>> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'`
>> will always stack overflow for either with or without my change.
>>
>> The issue looks like the stack overflow problem Valentyn has commented on
>> this issue: https://issues.apache.org/jira/browse/BEAM-8324
>>
>> I can't really tell what's happening behind the scenes. Like how a test
>> passes and fails when invoked with different tools? (Note: the test will be
>> skipped if dependencies are not setup)
>>
>>
>> On Fri, Oct 11, 2019 at 5:51 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> Silly question, but how did you install Beam? did you use pip install -e
>>> .[gcp,test]?
>>>
>>> On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
>>>
>>>> Thanks Valentyn, I've tried rebasing several times and caught up with
>>>> upstream several times from yesterday but the issue is not gone.
>>>> I can see the dill is pinned to 0.3.0. And this is true in my code base
>>>> and in clean beam repo (freshly cloned today).
>>>> I just don't know how the suite skipped/passed the tests at current
>>>> master. Because when the test is run, there is the failure.
>>>>
>>>> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>>> Try to rebase your branch on top of master (or just change this line
>>>>> [2] )
>>>>>
>>>>> There is an issue with dill==0.3.1.1 that we did not rootcause yet
>>>>> [1], but we have pinned dill to 0.3.0 on master for now.
>>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-5878
>>>>>
>>>>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>>>>
>>>>>
>>>>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>>>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>>>>
>>>>>> And I've also toyed with *tox.ini* to run the test with a clean beam
>>>>>> repo.
>>>>>> To run the tests:
>>>>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>>>>> To speed up the process by only running the test with the issue, you
>>>>>> can change the tox.ini from line 153:
>>>>>>
>>>>>> [testenv:py37-gcp]
>>>>>> setenv =
>>>>>> RUN_SKIPPED_PY3_TESTS=0
>>>>>> extras = test,gcp
>>>>>> commands =
>>>>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>>>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>>>>
>>>>>> It turns out this stack overflow error occurs even in a clean beam
>>>>>> repo if you just run the single test.
>>>>>> However, if you just run the original py37-gcp test (the commented
>>>>>> line), everything passes.
>>>>>> In that scenario, by checking the generated *nosetests.xml*, we can
>>>>>> see the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>>>>> message="GCP dependencies are not installed"><![CDATA[Exception: GCP
>>>>>> dependencies are not installed
>>>>>> However, py37-gcp does have gcp dependency and should not skip the
>>>>>> test.
>>>>>>
>>>>>> I'm wondering if the error exists long time ago. My change has
>>>>>> nothing to do with dataflow runner (in completely different packages and
>>>>>> new files are not referenced at all). But somehow, the change triggers the
>>>>>> test to be run. Similar to modifying the *tox.ini* file, that also
>>>>>> trigger the test to be run.
>>>>>>
>>>>>> Attached stack trace. Looks like _dill.py and pickle.py are calling
>>>>>> each other into an infinite loop.
>>>>>>
>>>>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>>>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error:
>>>>>> Cannot recover from stack overflow.
>>>>>>
>>>>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 479 in get
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 497 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1394 in save_function
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 885 in _batch_setitems
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 859 in save_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 910 in save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 198 in new_save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 114 in wrapper
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1137 in save_cell
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1394 in save_function
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 885 in _batch_setitems
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 859 in save_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 910 in save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 198 in new_save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 114 in wrapper
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1137 in save_cell
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1394 in save_function
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 885 in _batch_setitems
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 859 in save_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 910 in save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 198 in new_save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 114 in wrapper
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1137 in save_cell
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1394 in save_function
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 885 in _batch_setitems
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 859 in save_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 910 in save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 198 in new_save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 114 in wrapper
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1137 in save_cell
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 774 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 789 in save_tuple
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 638 in save_reduce
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 1394 in save_function
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 885 in _batch_setitems
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 859 in save_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>>> , line 910 in save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>>> , line 198 in new_save_module_dict
>>>>>>
>>>>>> File
>>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>>> , line 504 in save
>>>>>>
>>>>>> ...
>>>>>>
>>>>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/
>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-
>>>>>> gcp/py37-gcp/bin/python setup.py nosetests --tests apache_beam/
>>>>>> runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>>>>>> (exited with code -6)
>>>>>>
>>>>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/
>>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>>>>> run_tox_cleanup.sh
>>>>>>
>>>>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Valentyn Tymofieiev <va...@google.com>.
Thanks, I can also reproduce this error.
It appears that running this test through nose makes the test fail on
Python 3.7, however the test does not fail on its own, and does not fail in
a tox suite. However the test starts failing in tox if we upgrade to
dill==0.3.1.1, so I associated this failure with an upgrade to
dill==0.3.1.1 in the past. To summarize:
`python ./setup.py test -s
apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
passes.
`tox -e py37-gcp` passes if Beam depends on dill==0.3.0, but fails if Beam
depends on dill==0.3.1.1.
`python ./setup.py nosetests --tests
'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data`
fails currently if run on master.
I filed https://issues.apache.org/jira/browse/BEAM-8397 to investigate
this. You mentioned that you are currently blocked on this error - are you
not able to submit a PR or in what way are you blocked? My understanding is
that this error is not surfacing in precommit tests yet.
Thanks,
Valentyn
On Mon, Oct 14, 2019 at 11:48 AM Ning Kang <ni...@google.com> wrote:
> I tried installing with `pip install -e .[gcp,test]` locally from a cloned
> repo without my changes. Still no luck.
> And when running the task, there would be `setupVirtualenv` task to setup
> the dependencies, so the gcp dependency should be available.
> I don't think the dependency is the problem because otherwise, the test
> will be skipped and I won't run into the error.
> I've checked the `dill` installed is 0.3.0. Force re-installed
> dependencies, force rerun the tasks.
>
> The only time the test passes with gradle and tox for me is when I run the
> task `*:sdks:python:test-suites:tox:py37:testPy37Gcp*` including all the
> tests with `python setup.py nosetests --ignore-files '.*py3[8-9]\.py$'
> {posargs}` from a clean repo.
> Also, cmd `python ./setup.py test -s
> apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
> shows OK for me with and without my change.
> But if I use nosetests, cmd `python ./setup.py nosetests --tests
> 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'`
> will always stack overflow for either with or without my change.
>
> The issue looks like the stack overflow problem Valentyn has commented on
> this issue: https://issues.apache.org/jira/browse/BEAM-8324
>
> I can't really tell what's happening behind the scenes. Like how a test
> passes and fails when invoked with different tools? (Note: the test will be
> skipped if dependencies are not setup)
>
>
> On Fri, Oct 11, 2019 at 5:51 PM Pablo Estrada <pa...@google.com> wrote:
>
>> Silly question, but how did you install Beam? did you use pip install -e
>> .[gcp,test]?
>>
>> On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
>>
>>> Thanks Valentyn, I've tried rebasing several times and caught up with
>>> upstream several times from yesterday but the issue is not gone.
>>> I can see the dill is pinned to 0.3.0. And this is true in my code base
>>> and in clean beam repo (freshly cloned today).
>>> I just don't know how the suite skipped/passed the tests at current
>>> master. Because when the test is run, there is the failure.
>>>
>>> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <va...@google.com>
>>> wrote:
>>>
>>>> Try to rebase your branch on top of master (or just change this line
>>>> [2] )
>>>>
>>>> There is an issue with dill==0.3.1.1 that we did not rootcause yet [1],
>>>> but we have pinned dill to 0.3.0 on master for now.
>>>>
>>>> https://issues.apache.org/jira/browse/BEAM-5878
>>>>
>>>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>>>
>>>>
>>>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>>>
>>>>> And I've also toyed with *tox.ini* to run the test with a clean beam
>>>>> repo.
>>>>> To run the tests:
>>>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>>>> To speed up the process by only running the test with the issue, you
>>>>> can change the tox.ini from line 153:
>>>>>
>>>>> [testenv:py37-gcp]
>>>>> setenv =
>>>>> RUN_SKIPPED_PY3_TESTS=0
>>>>> extras = test,gcp
>>>>> commands =
>>>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>>>
>>>>> It turns out this stack overflow error occurs even in a clean beam
>>>>> repo if you just run the single test.
>>>>> However, if you just run the original py37-gcp test (the commented
>>>>> line), everything passes.
>>>>> In that scenario, by checking the generated *nosetests.xml*, we can
>>>>> see the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>>>> message="GCP dependencies are not installed"><![CDATA[Exception: GCP
>>>>> dependencies are not installed
>>>>> However, py37-gcp does have gcp dependency and should not skip the
>>>>> test.
>>>>>
>>>>> I'm wondering if the error exists long time ago. My change has nothing
>>>>> to do with dataflow runner (in completely different packages and new files
>>>>> are not referenced at all). But somehow, the change triggers the test to be
>>>>> run. Similar to modifying the *tox.ini* file, that also trigger the
>>>>> test to be run.
>>>>>
>>>>> Attached stack trace. Looks like _dill.py and pickle.py are calling
>>>>> each other into an infinite loop.
>>>>>
>>>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error:
>>>>> Cannot recover from stack overflow.
>>>>>
>>>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 479 in get
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 497 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1394 in save_function
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 885 in _batch_setitems
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 859 in save_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 910 in save_module_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 198 in new_save_module_dict
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 114 in wrapper
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1137 in save_cell
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1394 in save_function
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 885 in _batch_setitems
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 859 in save_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 910 in save_module_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 198 in new_save_module_dict
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 114 in wrapper
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1137 in save_cell
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1394 in save_function
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 885 in _batch_setitems
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 859 in save_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 910 in save_module_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 198 in new_save_module_dict
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 114 in wrapper
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1137 in save_cell
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1394 in save_function
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 885 in _batch_setitems
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 859 in save_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 910 in save_module_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 198 in new_save_module_dict
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 114 in wrapper
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1137 in save_cell
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 774 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 789 in save_tuple
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 638 in save_reduce
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 1394 in save_function
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 885 in _batch_setitems
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 859 in save_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>>> , line 910 in save_module_dict
>>>>>
>>>>> File
>>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>>> , line 198 in new_save_module_dict
>>>>>
>>>>> File
>>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>>> , line 504 in save
>>>>>
>>>>> ...
>>>>>
>>>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/python
>>>>> /test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37
>>>>> -gcp/bin/python setup.py nosetests --tests apache_beam/runners/
>>>>> dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>>>>> (exited with code -6)
>>>>>
>>>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/
>>>>> python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>>>> run_tox_cleanup.sh
>>>>>
>>>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Ning Kang <ni...@google.com>.
I tried installing with `pip install -e .[gcp,test]` locally from a cloned
repo without my changes. Still no luck.
And when running the task, there would be `setupVirtualenv` task to setup
the dependencies, so the gcp dependency should be available.
I don't think the dependency is the problem because otherwise, the test
will be skipped and I won't run into the error.
I've checked the `dill` installed is 0.3.0. Force re-installed
dependencies, force rerun the tasks.
The only time the test passes with gradle and tox for me is when I run the
task `*:sdks:python:test-suites:tox:py37:testPy37Gcp*` including all the
tests with `python setup.py nosetests --ignore-files '.*py3[8-9]\.py$'
{posargs}` from a clean repo.
Also, cmd `python ./setup.py test -s
apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest.test_remote_runner_display_data`
shows OK for me with and without my change.
But if I use nosetests, cmd `python ./setup.py nosetests --tests
'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data'`
will always stack overflow for either with or without my change.
The issue looks like the stack overflow problem Valentyn has commented on
this issue: https://issues.apache.org/jira/browse/BEAM-8324
I can't really tell what's happening behind the scenes. Like how a test
passes and fails when invoked with different tools? (Note: the test will be
skipped if dependencies are not setup)
On Fri, Oct 11, 2019 at 5:51 PM Pablo Estrada <pa...@google.com> wrote:
> Silly question, but how did you install Beam? did you use pip install -e
> .[gcp,test]?
>
> On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
>
>> Thanks Valentyn, I've tried rebasing several times and caught up with
>> upstream several times from yesterday but the issue is not gone.
>> I can see the dill is pinned to 0.3.0. And this is true in my code base
>> and in clean beam repo (freshly cloned today).
>> I just don't know how the suite skipped/passed the tests at current
>> master. Because when the test is run, there is the failure.
>>
>> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>>> Try to rebase your branch on top of master (or just change this line [2]
>>> )
>>>
>>> There is an issue with dill==0.3.1.1 that we did not rootcause yet [1],
>>> but we have pinned dill to 0.3.0 on master for now.
>>>
>>> https://issues.apache.org/jira/browse/BEAM-5878
>>>
>>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>>
>>>
>>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>>
>>>> And I've also toyed with *tox.ini* to run the test with a clean beam
>>>> repo.
>>>> To run the tests:
>>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>>> To speed up the process by only running the test with the issue, you
>>>> can change the tox.ini from line 153:
>>>>
>>>> [testenv:py37-gcp]
>>>> setenv =
>>>> RUN_SKIPPED_PY3_TESTS=0
>>>> extras = test,gcp
>>>> commands =
>>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>>
>>>> It turns out this stack overflow error occurs even in a clean beam repo
>>>> if you just run the single test.
>>>> However, if you just run the original py37-gcp test (the commented
>>>> line), everything passes.
>>>> In that scenario, by checking the generated *nosetests.xml*, we can
>>>> see the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>>> message="GCP dependencies are not installed"><![CDATA[Exception: GCP
>>>> dependencies are not installed
>>>> However, py37-gcp does have gcp dependency and should not skip the test.
>>>>
>>>> I'm wondering if the error exists long time ago. My change has nothing
>>>> to do with dataflow runner (in completely different packages and new files
>>>> are not referenced at all). But somehow, the change triggers the test to be
>>>> run. Similar to modifying the *tox.ini* file, that also trigger the
>>>> test to be run.
>>>>
>>>> Attached stack trace. Looks like _dill.py and pickle.py are calling
>>>> each other into an infinite loop.
>>>>
>>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot
>>>> recover from stack overflow.
>>>>
>>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 479 in get
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 497 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1394 in save_function
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 885 in _batch_setitems
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 859 in save_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 910 in save_module_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 198 in new_save_module_dict
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 114 in wrapper
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1137 in save_cell
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1394 in save_function
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 885 in _batch_setitems
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 859 in save_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 910 in save_module_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 198 in new_save_module_dict
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 114 in wrapper
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1137 in save_cell
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1394 in save_function
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 885 in _batch_setitems
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 859 in save_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 910 in save_module_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 198 in new_save_module_dict
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 114 in wrapper
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1137 in save_cell
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1394 in save_function
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 885 in _batch_setitems
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 859 in save_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 910 in save_module_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 198 in new_save_module_dict
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 114 in wrapper
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1137 in save_cell
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 774 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 789 in save_tuple
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 638 in save_reduce
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 1394 in save_function
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 885 in _batch_setitems
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 859 in save_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>>> , line 910 in save_module_dict
>>>>
>>>> File
>>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>>> , line 198 in new_save_module_dict
>>>>
>>>> File
>>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>>> , line 504 in save
>>>>
>>>> ...
>>>>
>>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/python/
>>>> test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-
>>>> gcp/bin/python setup.py nosetests --tests apache_beam/runners/dataflow/
>>>> dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>>>> (exited with code -6)
>>>>
>>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/
>>>> python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>>> run_tox_cleanup.sh
>>>>
>>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Pablo Estrada <pa...@google.com>.
Silly question, but how did you install Beam? did you use pip install -e
.[gcp,test]?
On Fri, Oct 11, 2019 at 5:46 PM Ning Kang <ni...@google.com> wrote:
> Thanks Valentyn, I've tried rebasing several times and caught up with
> upstream several times from yesterday but the issue is not gone.
> I can see the dill is pinned to 0.3.0. And this is true in my code base
> and in clean beam repo (freshly cloned today).
> I just don't know how the suite skipped/passed the tests at current
> master. Because when the test is run, there is the failure.
>
> On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> Try to rebase your branch on top of master (or just change this line [2] )
>>
>> There is an issue with dill==0.3.1.1 that we did not rootcause yet [1],
>> but we have pinned dill to 0.3.0 on master for now.
>>
>> https://issues.apache.org/jira/browse/BEAM-5878
>>
>> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>>
>>
>> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>>
>>> Hi everyone,
>>>
>>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>>
>>> And I've also toyed with *tox.ini* to run the test with a clean beam
>>> repo.
>>> To run the tests:
>>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>>> To speed up the process by only running the test with the issue, you can
>>> change the tox.ini from line 153:
>>>
>>> [testenv:py37-gcp]
>>> setenv =
>>> RUN_SKIPPED_PY3_TESTS=0
>>> extras = test,gcp
>>> commands =
>>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>>
>>> It turns out this stack overflow error occurs even in a clean beam repo
>>> if you just run the single test.
>>> However, if you just run the original py37-gcp test (the commented
>>> line), everything passes.
>>> In that scenario, by checking the generated *nosetests.xml*, we can see
>>> the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>>> message="GCP dependencies are not installed"><![CDATA[Exception: GCP
>>> dependencies are not installed
>>> However, py37-gcp does have gcp dependency and should not skip the test.
>>>
>>> I'm wondering if the error exists long time ago. My change has nothing
>>> to do with dataflow runner (in completely different packages and new files
>>> are not referenced at all). But somehow, the change triggers the test to be
>>> run. Similar to modifying the *tox.ini* file, that also trigger the
>>> test to be run.
>>>
>>> Attached stack trace. Looks like _dill.py and pickle.py are calling each
>>> other into an infinite loop.
>>>
>>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot
>>> recover from stack overflow.
>>>
>>> Current thread 0x0000000112f645c0 (most recent call first):
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 479 in get
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 497 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1394 in save_function
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 885 in _batch_setitems
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 859 in save_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 910 in save_module_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 198 in new_save_module_dict
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 114 in wrapper
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1137 in save_cell
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1394 in save_function
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 885 in _batch_setitems
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 859 in save_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 910 in save_module_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 198 in new_save_module_dict
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 114 in wrapper
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1137 in save_cell
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1394 in save_function
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 885 in _batch_setitems
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 859 in save_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 910 in save_module_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 198 in new_save_module_dict
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 114 in wrapper
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1137 in save_cell
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1394 in save_function
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 885 in _batch_setitems
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 859 in save_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 910 in save_module_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 198 in new_save_module_dict
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 114 in wrapper
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1137 in save_cell
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 774 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 789 in save_tuple
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 638 in save_reduce
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 1394 in save_function
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 885 in _batch_setitems
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 859 in save_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>>> , line 910 in save_module_dict
>>>
>>> File
>>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>>> , line 198 in new_save_module_dict
>>>
>>> File
>>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>>> , line 504 in save
>>>
>>> ...
>>>
>>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/python/
>>> test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-
>>> gcp/bin/python setup.py nosetests --tests apache_beam/runners/dataflow/
>>> dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>>> (exited with code -6)
>>>
>>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/
>>> python/test-suites/tox/py37/build/srcs/sdks/python/scripts/
>>> run_tox_cleanup.sh
>>>
>>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Ning Kang <ni...@google.com>.
Thanks Valentyn, I've tried rebasing several times and caught up with
upstream several times from yesterday but the issue is not gone.
I can see the dill is pinned to 0.3.0. And this is true in my code base and
in clean beam repo (freshly cloned today).
I just don't know how the suite skipped/passed the tests at current master.
Because when the test is run, there is the failure.
On Fri, Oct 11, 2019 at 5:02 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> Try to rebase your branch on top of master (or just change this line [2] )
>
> There is an issue with dill==0.3.1.1 that we did not rootcause yet [1],
> but we have pinned dill to 0.3.0 on master for now.
>
> https://issues.apache.org/jira/browse/BEAM-5878
>
> https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
>
>
> On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
>
>> Hi everyone,
>>
>> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
>> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>>
>> And I've also toyed with *tox.ini* to run the test with a clean beam
>> repo.
>> To run the tests:
>> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
>> To speed up the process by only running the test with the issue, you can
>> change the tox.ini from line 153:
>>
>> [testenv:py37-gcp]
>> setenv =
>> RUN_SKIPPED_PY3_TESTS=0
>> extras = test,gcp
>> commands =
>> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
>> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>>
>> It turns out this stack overflow error occurs even in a clean beam repo
>> if you just run the single test.
>> However, if you just run the original py37-gcp test (the commented line),
>> everything passes.
>> In that scenario, by checking the generated *nosetests.xml*, we can see
>> the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
>> message="GCP dependencies are not installed"><![CDATA[Exception: GCP
>> dependencies are not installed
>> However, py37-gcp does have gcp dependency and should not skip the test.
>>
>> I'm wondering if the error exists long time ago. My change has nothing to
>> do with dataflow runner (in completely different packages and new files are
>> not referenced at all). But somehow, the change triggers the test to be
>> run. Similar to modifying the *tox.ini* file, that also trigger the test
>> to be run.
>>
>> Attached stack trace. Looks like _dill.py and pickle.py are calling each
>> other into an infinite loop.
>>
>> test_remote_runner_display_data (apache_beam.runners.dataflow.
>> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot
>> recover from stack overflow.
>>
>> Current thread 0x0000000112f645c0 (most recent call first):
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 479 in get
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 497 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1394 in save_function
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 885 in _batch_setitems
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 859 in save_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 910 in save_module_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 198 in new_save_module_dict
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 114 in wrapper
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1137 in save_cell
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1394 in save_function
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 885 in _batch_setitems
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 859 in save_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 910 in save_module_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 198 in new_save_module_dict
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 114 in wrapper
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1137 in save_cell
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1394 in save_function
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 885 in _batch_setitems
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 859 in save_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 910 in save_module_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 198 in new_save_module_dict
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 114 in wrapper
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1137 in save_cell
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1394 in save_function
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 885 in _batch_setitems
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 859 in save_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 910 in save_module_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 198 in new_save_module_dict
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 114 in wrapper
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1137 in save_cell
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 774 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 789 in save_tuple
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 638 in save_reduce
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 1394 in save_function
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 885 in _batch_setitems
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 859 in save_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
>> , line 910 in save_module_dict
>>
>> File
>> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
>> , line 198 in new_save_module_dict
>>
>> File
>> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
>> , line 504 in save
>>
>> ...
>>
>> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/python/
>> test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp
>> /bin/python setup.py nosetests --tests apache_beam/runners/dataflow/
>> dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
>> (exited with code -6)
>>
>> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/python
>> /test-suites/tox/py37/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
>>
>
Re: Fatal Python error: Cannot recover from stack overflow
Posted by Valentyn Tymofieiev <va...@google.com>.
Try to rebase your branch on top of master (or just change this line [2] )
There is an issue with dill==0.3.1.1 that we did not rootcause yet [1], but
we have pinned dill to 0.3.0 on master for now.
https://issues.apache.org/jira/browse/BEAM-5878
https://github.com/apache/beam/blob/2ff462da2e9304e904983a7cd747d31526b593c3/sdks/python/setup.py#L109
On Fri, Oct 11, 2019 at 4:55 PM Ning Kang <ni...@google.com> wrote:
> Hi everyone,
>
> I'm blocked by a stack overflow error from Python test: test_remote_runner_display_data
> (apache_beam.runners.dataflow.dataflow_runner_test.DataflowRunnerTest).
>
> And I've also toyed with *tox.ini* to run the test with a clean beam repo.
> To run the tests:
> *gradle :sdks:python:test-suites:tox:py37:testPy37Gcp*
> To speed up the process by only running the test with the issue, you can
> change the tox.ini from line 153:
>
> [testenv:py37-gcp]
> setenv =
> RUN_SKIPPED_PY3_TESTS=0
> extras = test,gcp
> commands =
> # python setup.py nosetests --ignore-files '.*py3[8-9]\.py$' {posargs}
> python setup.py nosetests --tests 'apache_beam/runners/dataflow/dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data' {posargs}
>
> It turns out this stack overflow error occurs even in a clean beam repo if
> you just run the single test.
> However, if you just run the original py37-gcp test (the commented line),
> everything passes.
> In that scenario, by checking the generated *nosetests.xml*, we can see
> the test throwing error is skipped: <skipped type="unittest.case.SkipTest"
> message="GCP dependencies are not installed"><![CDATA[Exception: GCP
> dependencies are not installed
> However, py37-gcp does have gcp dependency and should not skip the test.
>
> I'm wondering if the error exists long time ago. My change has nothing to
> do with dataflow runner (in completely different packages and new files are
> not referenced at all). But somehow, the change triggers the test to be
> run. Similar to modifying the *tox.ini* file, that also trigger the test
> to be run.
>
> Attached stack trace. Looks like _dill.py and pickle.py are calling each
> other into an infinite loop.
>
> test_remote_runner_display_data (apache_beam.runners.dataflow.
> dataflow_runner_test.DataflowRunnerTest) ... Fatal Python error: Cannot
> recover from stack overflow.
>
> Current thread 0x0000000112f645c0 (most recent call first):
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 479 in get
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 497 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1394 in save_function
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 885 in _batch_setitems
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 859 in save_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 910 in save_module_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 198 in new_save_module_dict
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 114 in wrapper
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1137 in save_cell
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1394 in save_function
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 885 in _batch_setitems
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 859 in save_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 910 in save_module_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 198 in new_save_module_dict
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 114 in wrapper
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1137 in save_cell
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1394 in save_function
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 885 in _batch_setitems
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 859 in save_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 910 in save_module_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 198 in new_save_module_dict
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 114 in wrapper
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1137 in save_cell
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1394 in save_function
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 885 in _batch_setitems
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 859 in save_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 910 in save_module_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 198 in new_save_module_dict
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 114 in wrapper
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1137 in save_cell
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 774 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 789 in save_tuple
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 638 in save_reduce
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 1394 in save_function
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 885 in _batch_setitems
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 859 in save_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/lib/python3.7/site-packages/dill/_dill.py"
> , line 910 in save_module_dict
>
> File
> "/Users/ningk/repos/beam/sdks/python/test-suites/tox/py37/build/srcs/sdks/python/apache_beam/internal/pickler.py"
> , line 198 in new_save_module_dict
>
> File
> "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py"
> , line 504 in save
>
> ...
>
> ERROR: InvocationError for command /Users/ningk/repos/beam/sdks/python/
> test-suites/tox/py37/build/srcs/sdks/python/target/.tox-py37-gcp/py37-gcp/
> bin/python setup.py nosetests --tests apache_beam/runners/dataflow/
> dataflow_runner_test.py:DataflowRunnerTest.test_remote_runner_display_data
> (exited with code -6)
>
> py37-gcp run-test-post: commands[0] | /Users/ningk/repos/beam/sdks/python/
> test-suites/tox/py37/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
>