You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Cabell Maddux (Jira)" <ji...@apache.org> on 2019/12/03 17:37:00 UTC
[jira] [Updated] (AIRFLOW-6166) KubernetesPodOperator resolve
default xcom read behavior or improve associated documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cabell Maddux updated AIRFLOW-6166:
-----------------------------------
Description:
Hi,
I'm relatively new to airflow and am testing out Airflow running with the Kubernetes executor locally on Minikube based off of the CI configs in the repo ([https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes/app]).
Was able to get everything up and running smoothly, however thought default behavior for xcom when no `do_xcom_push` argument is provided to a `KubernetesPodOperator` was a bit confusing.
If a user doesn't explicitly create a result.json in the pod default `XCOM_MOUNT_PATH` as part of the task pod, obviously the task will fail ([https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_launcher.py#L214)]
```
cat: can't open '/airflow/xcom/return.json': No such file or directory
```
Rather than the user having to explicitly create result.json as part of their task, maybe more aligned with other operators to me to create `\{XCOM_MOUNT_PATH}/result.json` when creating the xcom sidecar - https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_generator.py#L36.
Otherwise, maybe would be nice to document that user of KubernetesPodOperator should be careful to explicitly deal with `do_xcom_push` to avoid a bit of confusion.
was:
Hi,
I'm relatively new to airflow and am testing out Airflow running with the Kubernetes executor locally on Minikube based off of the CI configs in the repo ([https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes/app]).
Was able to get everything up and running smoothly, however the default behavior for xcom when no `do_xcom_push` argument is provided to a `KubernetesPodOperator` is a bit confusing.
If a user doesn't explicitly create a result.json in the pod default `XCOM_MOUNT_PATH` as part of the task pod, the task will fail ([https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_launcher.py#L214)]
```
```
Rather than the user having to explicitly create result.json as part of their task, seems more aligned with other operators to me to create `\{XCOM_MOUNT_PATH}/result.json` when creating the
> KubernetesPodOperator resolve default xcom read behavior or improve associated documentation
> --------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-6166
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6166
> Project: Apache Airflow
> Issue Type: Improvement
> Components: documentation, executor-kubernetes, xcom
> Affects Versions: 1.10.7
> Reporter: Cabell Maddux
> Assignee: Daniel Imberman
> Priority: Minor
>
> Hi,
> I'm relatively new to airflow and am testing out Airflow running with the Kubernetes executor locally on Minikube based off of the CI configs in the repo ([https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes/app]).
> Was able to get everything up and running smoothly, however thought default behavior for xcom when no `do_xcom_push` argument is provided to a `KubernetesPodOperator` was a bit confusing.
> If a user doesn't explicitly create a result.json in the pod default `XCOM_MOUNT_PATH` as part of the task pod, obviously the task will fail ([https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_launcher.py#L214)]
> ```
> cat: can't open '/airflow/xcom/return.json': No such file or directory
> ```
> Rather than the user having to explicitly create result.json as part of their task, maybe more aligned with other operators to me to create `\{XCOM_MOUNT_PATH}/result.json` when creating the xcom sidecar - https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_generator.py#L36.
> Otherwise, maybe would be nice to document that user of KubernetesPodOperator should be careful to explicitly deal with `do_xcom_push` to avoid a bit of confusion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)