You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Valentyn Tymofieiev (Jira)" <ji...@apache.org> on 2019/09/30 17:02:00 UTC
[jira] [Comment Edited] (BEAM-8324) Pre/Postcommit Dataflow IT
tests fail: _create_function() takes from 2 to 6 positional arguments but 7
were given
[ https://issues.apache.org/jira/browse/BEAM-8324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941140#comment-16941140 ]
Valentyn Tymofieiev edited comment on BEAM-8324 at 9/30/19 5:01 PM:
--------------------------------------------------------------------
I think the cause of the error is that we pickle main session using dill==0.3.1.1, and unpickle it using dill==0.3.0 (which is installed on DF workers).
We can update the version of Dill in DF workers, or restrict the upperbound for dill, which may be advisable given [~yoshiki.obata]'s report in https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16941114&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16941114.
was (Author: tvalentyn):
I think the cause of the error is that we pickle main session using dill==0.3.1.1, and unpickle it using dill==0.2.9 (which is installed on DF workers).
We can update the version of Dill in DF workers, or restrict the upperbound for dill, which may be advisable given [~yoshiki.obata]'s report in https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16941114&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16941114.
> Pre/Postcommit Dataflow IT tests fail: _create_function() takes from 2 to 6 positional arguments but 7 were given
> -----------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-8324
> URL: https://issues.apache.org/jira/browse/BEAM-8324
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Fix For: 2.16.0
>
>
> {noformat}
> Dataflow pipeline failed. State: FAILED, Error:
> Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 773, in run
> self._load_main_session(self.local_staging_directory)
> File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 489, in _load_main_session
> pickler.load_session(session_file)
> File "/usr/local/lib/python3.7/site-packages/apache_beam/internal/pickler.py", line 287, in load_session
> return dill.load_session(file_path)
> File "/usr/local/lib/python3.7/site-packages/dill/_dill.py", line 410, in load_session
> module = unpickler.load()
> TypeError: _create_function() takes from 2 to 6 positional arguments but 7 were given
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)