You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Adam C Baker (JIRA)" <ji...@apache.org> on 2019/01/02 18:24:00 UTC
[jira] [Updated] (AIRFLOW-3620) Inlets and outlets are always empty
in bash operator templates
[ https://issues.apache.org/jira/browse/AIRFLOW-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam C Baker updated AIRFLOW-3620:
----------------------------------
Description:
When creating a data pipeline where one task's input is its upstream task's output, it seems the only way to automatically coordinate these is to use the lineage feature and set {{inlets = \{"auto": True}}}. Doing this with a PythonOperator allows one to get the input and output data sources for the task by passing in the context and getting {{task.inlets}} and {{task.outlets}} values.
This fails with the BashOperator. With a template including something like {{{{task.inlets[0]}}}}, it throws an exception, and templating with {{{\{task.inlets}}}} or {{{\{task.outlets}}}} always reveals these values to be an empty list.
was:
When creating a data pipeline where one task's input is its upstream task's output, it seems the only way to automatically coordinate these is to use the lineage feature and set `inlets = \{"auto": True}`. Doing this with a PythonOperator allows one to get the input and output data sources for the task by passing in the context and getting `task.inlets` and `task.outlets` values.
This fails with the BashOperator. With a template including something like `\{{task.inlets[0]}}`, it throws an exception, and templating with `\{{task.inlets}}` or `\{{task.outlets}}` always reveals these values to be an empty list.
> Inlets and outlets are always empty in bash operator templates
> --------------------------------------------------------------
>
> Key: AIRFLOW-3620
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3620
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.10.1
> Reporter: Adam C Baker
> Priority: Major
>
> When creating a data pipeline where one task's input is its upstream task's output, it seems the only way to automatically coordinate these is to use the lineage feature and set {{inlets = \{"auto": True}}}. Doing this with a PythonOperator allows one to get the input and output data sources for the task by passing in the context and getting {{task.inlets}} and {{task.outlets}} values.
> This fails with the BashOperator. With a template including something like {{{{task.inlets[0]}}}}, it throws an exception, and templating with {{{\{task.inlets}}}} or {{{\{task.outlets}}}} always reveals these values to be an empty list.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)