You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Eugene Serdyuk (Jira)" <ji...@apache.org> on 2019/12/11 17:37:00 UTC
[jira] [Comment Edited] (AIRFLOW-6217) KubernetesPodOperator XCom
pushes not working
[ https://issues.apache.org/jira/browse/AIRFLOW-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993742#comment-16993742 ]
Eugene Serdyuk edited comment on AIRFLOW-6217 at 12/11/19 5:36 PM:
-------------------------------------------------------------------
Finally got it working even on 1.11.10. I've just reinstalled minikube cluster and changed postgresql helm chart dependency to the newer version (also postgres updated from 9.6.* to 11.7.0).
Unit tests made for xcom pushing are now working fine.
It also turned out for me that to xcom_push a KubernetesPodOperator (KPO) results, you have to do it manually via cmds/arguments parameters.
In our project we are using factory pattern to create KPO's, and to xcom_push correctly it became needed to decorate this operator's arguments attribute by adding the following line:
{code:java}
' | python -c "import json, sys, os; os.makedirs(\'/airflow/xcom\', exist_ok=True); f = open(\'/airflow/xcom/return.json\', \'w\'); json.dump({\'result\': sys.stdin.read()}, f); f.close()"'{code}
For me it looks VERY inconvenient.
was (Author: eserdk):
Finally got it working even on 1.11.10. I've just reinstalled minikube cluster and changed postgresql helm chart dependency to the newer version (also postgres updated from 9.6.* to 11.7.0).
Unit tests made for xcom pushing are now working fine.
It also turned out for me that to xcom_push a KubernetesPodOperator (KPO) results, you have to do it manually via cmds/arguments parameters.
In our project we are using factory pattern for creating KPO's and to xcom_push correctly it became needed to decorate this operator's arguments attribute by adding the following line:
{code:java}
' | python -c "import json, sys, os; os.makedirs(\'/airflow/xcom\', exist_ok=True); f = open(\'/airflow/xcom/return.json\', \'w\'); json.dump({\'result\': sys.stdin.read()}, f); f.close()"'{code}
For me it looks VERY inconvenient.
> KubernetesPodOperator XCom pushes not working
> ---------------------------------------------
>
> Key: AIRFLOW-6217
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6217
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, xcom
> Affects Versions: 1.10.6
> Environment: Kubernetes version: 1.11.10
> Minikube version: 1.5.2
> Airflow version: 1.10.6
> Reporter: Eugene Serdyuk
> Priority: Major
>
>
> XCom pushes don’t work with KubernetesPodOperator both when I’m using LocalExecutor and KubernetesExecutor.
> I do write a return information to the /airflow/xcom/return.json, but despite this fact it’s still an error:
>
> {code:java}
> [2019-12-06 15:12:40,116] {logging_mixin.py:112} INFO - [2019-12-06 15:12:40,116] {pod_launcher.py:217} INFO - Running command... cat /airflow/xcom/return.json
> [2019-12-06 15:12:40,201] {logging_mixin.py:112} INFO - [2019-12-06 15:12:40,201] {pod_launcher.py:224} INFO - cat: can't open '/airflow/xcom/return.json': No such file or directory{code}
>
> I've also implemented the same code that is written [here|https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L315].
> But this error still persists. In other words, this test doest not pass.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)