You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Yaroslav Mykolayovysh Kolodiy (Jira)" <ji...@apache.org> on 2019/12/02 14:47:00 UTC
[jira] [Comment Edited] (AIRFLOW-2738) XCOM's don't work with
PythonVirtualenvOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986080#comment-16986080 ]
Yaroslav Mykolayovysh Kolodiy edited comment on AIRFLOW-2738 at 12/2/19 2:46 PM:
---------------------------------------------------------------------------------
Same problem in version 1.10.6, simple [example|[https://github.com/apache/airflow/blob/dd175fa8db222dad763ad35cedf85a750b52d4fb/airflow/example_dags/example_xcom.py]] but changing it to PythonVirtualenvOperator (with correct parameters).
Any progress?
h1.
was (Author: yk0l0diy):
Same problem in version 1.10.6, simple example is using the pythonOperator but changing it to PythonVirtualenvOperator (with correct parameters).
Any progress?
h1.
> XCOM's don't work with PythonVirtualenvOperator
> -----------------------------------------------
>
> Key: AIRFLOW-2738
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2738
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.8.0, 1.9.0
> Reporter: Cezary
> Priority: Major
>
> When you set _provide_context_ to True in PythonVirtualenvOperator, and try to use xcom_push, Airflow is trying to pickle context - and throws error. __
> {code:java}
> Traceback (most recent call last):
> File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
> result = task_copy.execute(context=context)
> File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 93, in execute
> return_value = self.execute_callable()
> File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 249, in execute_callable
> self._write_args(input_filename)
> File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 305, in _write_args
> pickle.dump(arg_dict, f)
> TypeError: can't pickle module objects
> {code}
> I tried to debug - and it seems that _macros_, _task_instance_, _ti_ and _conf_ in context are guilty in this case.
> Using dill doesn't fix this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)