[jira] [Commented] (AIRFLOW-2738) XCOM's don't work with PythonVirtualenvOperator

["Yaroslav Mykolayovysh Kolodiy (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986080#comment-16986080 ] 

Yaroslav Mykolayovysh Kolodiy commented on AIRFLOW-2738:
--------------------------------------------------------

Same problem in version  1.10.6, simple example is using the pythonOperator but changing it to PythonVirtualenvOperator (with correct parameters).

 

Any progress?
h1.  

> XCOM's don't work with PythonVirtualenvOperator
> -----------------------------------------------
>
>                 Key: AIRFLOW-2738
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2738
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: Cezary
>            Priority: Major
>
> When you set _provide_context_ to True in PythonVirtualenvOperator, and try to use xcom_push, Airflow is trying to pickle context - and throws error. __ 
> {code:java}
> Traceback (most recent call last):
>   File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/models.py", line 1493, in _run_raw_task
>     result = task_copy.execute(context=context)
>   File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 93, in execute
>     return_value = self.execute_callable()
>   File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 249, in execute_callable
>     self._write_args(input_filename)
>   File "/home/czarek/.virtualenvs/airflow/lib/python3.6/site-packages/airflow/operators/python_operator.py", line 305, in _write_args
>     pickle.dump(arg_dict, f)
> TypeError: can't pickle module objects
> {code}
> I tried to debug - and it seems that _macros_, _task_instance_, _ti_ and _conf_  in context are guilty in this case.
> Using dill doesn't fix this problem.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)