You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Udi Meiri <eh...@google.com> on 2019/08/23 18:56:32 UTC
Python question about save_main_session
Hi,
I'm trying to get pytest with the xdist plugin to run Beam tests. The issue
is with save_main_session and a dependency of pytest-xdist called execnet,
which triggers this error:
*apache_beam/examples/complete/tfidf.py*:212: in run* output |
'write' >> WriteToText(known_args.output)**apache_beam/pipeline.py*:426:
in __exit__*
self.run().wait_until_finish()**apache_beam/pipeline.py*:406: in run*
self._options).run(False)**apache_beam/pipeline.py*:416: in run*
pickler.dump_session(os.path.join(tmpdir,
'main_session.pickle'))**apache_beam/internal/pickler.py*:282: in
dump_session*
dill.load_session(file_path)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/dill/_dill.py*:410:
in load_session* module =
unpickler.load()**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130:
in __getattr__* locs =
self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130:
in __getattr__* locs =
self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130:
in __getattr__* locs = self._importdef.get(name)**E
RecursionError: maximum recursion depth exceeded*
!!! Recursion detected (same locals & position)
Does anyone on this list have experience with these kinds of errors? Any
workarounds I can use? (can we handle this module specially / can we
exclude it from main session?)
Re: Python question about save_main_session
Posted by Chad Dombrova <ch...@gmail.com>.
Another solution may be writing a custom dill reducer for the object that
is triggering this recursion.
-chad
On Fri, Aug 23, 2019 at 6:52 PM Udi Meiri <eh...@google.com> wrote:
> Yeah that sounds like the sanest way forward.
> Lots of these tests are running standalone examples so they might need SMS
> if you run them directly.
> (example streaming_wordcount_it_test.py running streaming_wordcount.py)
>
> On Fri, Aug 23, 2019 at 3:17 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> I suggest re-writing the test to avoid save_main_session.
>>
>> On Fri, Aug 23, 2019 at 11:57 AM Udi Meiri <eh...@google.com> wrote:
>>
>>> Hi,
>>> I'm trying to get pytest with the xdist plugin to run Beam tests. The
>>> issue is with save_main_session and a dependency of pytest-xdist called
>>> execnet, which triggers this error:
>>>
>>> *apache_beam/examples/complete/tfidf.py*:212: in run* output | 'write' >> WriteToText(known_args.output)**apache_beam/pipeline.py*:426: in __exit__* self.run().wait_until_finish()**apache_beam/pipeline.py*:406: in run* self._options).run(False)**apache_beam/pipeline.py*:416: in run* pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))**apache_beam/internal/pickler.py*:282: in dump_session* dill.load_session(file_path)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/dill/_dill.py*:410: in load_session* module = unpickler.load()**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**E RecursionError: maximum recursion depth exceeded*
>>> !!! Recursion detected (same locals & position)
>>>
>>>
>>> Does anyone on this list have experience with these kinds of errors? Any
>>> workarounds I can use? (can we handle this module specially / can we
>>> exclude it from main session?)
>>>
>>
Re: Python question about save_main_session
Posted by Udi Meiri <eh...@google.com>.
Yeah that sounds like the sanest way forward.
Lots of these tests are running standalone examples so they might need SMS
if you run them directly.
(example streaming_wordcount_it_test.py running streaming_wordcount.py)
On Fri, Aug 23, 2019 at 3:17 PM Robert Bradshaw <ro...@google.com> wrote:
> I suggest re-writing the test to avoid save_main_session.
>
> On Fri, Aug 23, 2019 at 11:57 AM Udi Meiri <eh...@google.com> wrote:
>
>> Hi,
>> I'm trying to get pytest with the xdist plugin to run Beam tests. The
>> issue is with save_main_session and a dependency of pytest-xdist called
>> execnet, which triggers this error:
>>
>> *apache_beam/examples/complete/tfidf.py*:212: in run* output | 'write' >> WriteToText(known_args.output)**apache_beam/pipeline.py*:426: in __exit__* self.run().wait_until_finish()**apache_beam/pipeline.py*:406: in run* self._options).run(False)**apache_beam/pipeline.py*:416: in run* pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))**apache_beam/internal/pickler.py*:282: in dump_session* dill.load_session(file_path)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/dill/_dill.py*:410: in load_session* module = unpickler.load()**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**E RecursionError: maximum recursion depth exceeded*
>> !!! Recursion detected (same locals & position)
>>
>>
>> Does anyone on this list have experience with these kinds of errors? Any
>> workarounds I can use? (can we handle this module specially / can we
>> exclude it from main session?)
>>
>
Re: Python question about save_main_session
Posted by Robert Bradshaw <ro...@google.com>.
I suggest re-writing the test to avoid save_main_session.
On Fri, Aug 23, 2019 at 11:57 AM Udi Meiri <eh...@google.com> wrote:
> Hi,
> I'm trying to get pytest with the xdist plugin to run Beam tests. The
> issue is with save_main_session and a dependency of pytest-xdist called
> execnet, which triggers this error:
>
> *apache_beam/examples/complete/tfidf.py*:212: in run* output | 'write' >> WriteToText(known_args.output)**apache_beam/pipeline.py*:426: in __exit__* self.run().wait_until_finish()**apache_beam/pipeline.py*:406: in run* self._options).run(False)**apache_beam/pipeline.py*:416: in run* pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))**apache_beam/internal/pickler.py*:282: in dump_session* dill.load_session(file_path)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/dill/_dill.py*:410: in load_session* module = unpickler.load()**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**../../../../virtualenvs/beam-py35/lib/python3.5/site-packages/execnet/gateway_base.py*:130: in __getattr__* locs = self._importdef.get(name)**E RecursionError: maximum recursion depth exceeded*
> !!! Recursion detected (same locals & position)
>
>
> Does anyone on this list have experience with these kinds of errors? Any
> workarounds I can use? (can we handle this module specially / can we
> exclude it from main session?)
>