You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Ahmet Altay (Jira)" <ji...@apache.org> on 2019/09/30 16:57:00 UTC
[jira] [Created] (BEAM-8326) Run Portable_Python PreCommit fails with pickling

Ahmet Altay created BEAM-8326:
---------------------------------

             Summary: Run Portable_Python PreCommit fails with pickling
                 Key: BEAM-8326
                 URL: https://issues.apache.org/jira/browse/BEAM-8326
             Project: Beam
          Issue Type: Bug
          Components: sdk-py-core
            Reporter: Ahmet Altay
            Assignee: Valentyn Tymofieiev


Test log: https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/434/console

I see the following error:

09:31:34 Traceback (most recent call last):
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", line 261, in loads
09:31:34     return dill.loads(s)
09:31:34   File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 317, in loads
09:31:34     return load(file, ignore)
09:31:34   File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 305, in load
09:31:34     obj = pik.load()
09:31:34 TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were given
09:31:34 
09:31:34 During handling of the above exception, another exception occurred:
09:31:34 
09:31:34 Traceback (most recent call last):
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py", line 158, in _execute
09:31:34     response = task()
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py", line 191, in <lambda>
09:31:34     self._execute(lambda: worker.do_instruction(work), work)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py", line 343, in do_instruction
09:31:34     request.instruction_id)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py", line 363, in process_bundle
09:31:34     instruction_id, request.process_bundle_descriptor_id)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/sdk_worker.py", line 306, in get
09:31:34     self.data_channel_factory)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 580, in __init__
09:31:34     self.ops = self.create_execution_tree(self.process_bundle_descriptor)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 624, in create_execution_tree
09:31:34     descriptor.transforms, key=topological_height, reverse=True)])
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 623, in <listcomp>
09:31:34     for transform_id in sorted(
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 548, in wrapper
09:31:34     result = cache[args] = func(*args)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 607, in get_operation
09:31:34     in descriptor.transforms[transform_id].outputs.items()
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 606, in <dictcomp>
09:31:34     for tag, pcoll_id
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 605, in <listcomp>
09:31:34     tag: [get_operation(op) for op in pcoll_consumers[pcoll_id]]
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 548, in wrapper
09:31:34     result = cache[args] = func(*args)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 610, in get_operation
09:31:34     transform_id, transform_consumers)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 869, in create_operation
09:31:34     return creator(self, transform_id, transform_proto, payload, consumers)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1112, in create
09:31:34     serialized_fn, parameter)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1150, in _create_pardo_operation
09:31:34     dofn_data = pickler.loads(serialized_fn)
09:31:34   File "/usr/local/lib/python3.5/site-packages/apache_beam/internal/pickler.py", line 265, in loads
09:31:34     return dill.loads(s)
09:31:34   File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 317, in loads
09:31:34     return load(file, ignore)
09:31:34   File "/usr/local/lib/python3.5/site-packages/dill/_dill.py", line 305, in load
09:31:34     obj = pik.load()
09:31:34 TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were given

First exception `IndexError` is expected and handled here (https://github.com/apache/beam/blob/ee45cb03f6e6f3685ae0dccf75a8af4d259b3b1a/sdks/python/apache_beam/runners/worker/sdk_worker.py#L301)

The issue here is with pickling, might be a dill issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)