You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Chao-Han Tsai (JIRA)" <ji...@apache.org> on 2018/05/23 20:04:00 UTC
[jira] [Updated] (AIRFLOW-2517) backfill support passing key values
through CLI
[ https://issues.apache.org/jira/browse/AIRFLOW-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao-Han Tsai updated AIRFLOW-2517:
-----------------------------------
Description:
In backfill, we can provide key-value pairs through CLI and those pairs can be accessed through macros. This is just like the way `trigger_dag -c` works [1].
Let's walk through an example.
In the airflow CLI we specify a key-value pair.
{code:java}
airflow backfill hello_world -s 2018-02-01 -e 2018-02-08 -c ' {"text": "some text"}
{code}
In the DAG file, I have a `BashOperator` that contains a template command and I want
\{{ dag_run.conf.text }} resolves to the text I passed in CLI.
{code}
templated_command = """
echo "ds = {{ ds }}"
echo "prev_ds = {{ macros.datetime.strftime(prev_execution_date, "%Y-%m-%d") }}"
echo "next_ds = {{ macros.datetime.strftime(next_execution_date, "%Y-%m-%d") }}"
echo "text_through_conf = {{ dag_run.conf.text }}"
"""
bash_operator = BashOperator(
task_id='bash_task',
bash_command=templated_command,
dag=dag
)
{code}
[1] [https://airflow.apache.org/cli.html#trigger_dag]
was:
In backfill, we can provide key-value pairs through CLI and those pairs can be accessed through macros. This is just like the way `trigger_dag -c` works [1].
Let's walk through an example.
In the airflow CLI we specify a key-value pair.
{code}
airflow backfill hello_world -s 2018-02-01 -e 2018-02-08 -c ' {"text": "some text"}
{code}
```
In the DAG file, I have a `BashOperator` that contains a template command and I want
\{{ dag_run.conf.text }} resolves to the text I passed in CLI.
{code:python}
templated_command = """
echo "ds = {{ ds }}"
echo "prev_ds = {{ macros.datetime.strftime(prev_execution_date, "%Y-%m-%d") }}"
echo "next_ds = {{ macros.datetime.strftime(next_execution_date, "%Y-%m-%d") }}"
echo "text_through_conf = {{ dag_run.conf.text }}"
"""
bash_operator = BashOperator(
task_id='bash_task',
bash_command=templated_command,
dag=dag
)
{code}
[1] [https://airflow.apache.org/cli.html#trigger_dag]
> backfill support passing key values through CLI
> -----------------------------------------------
>
> Key: AIRFLOW-2517
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2517
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Chao-Han Tsai
> Assignee: Chao-Han Tsai
> Priority: Major
>
> In backfill, we can provide key-value pairs through CLI and those pairs can be accessed through macros. This is just like the way `trigger_dag -c` works [1].
> Let's walk through an example.
> In the airflow CLI we specify a key-value pair.
> {code:java}
> airflow backfill hello_world -s 2018-02-01 -e 2018-02-08 -c ' {"text": "some text"}
> {code}
>
> In the DAG file, I have a `BashOperator` that contains a template command and I want
> \{{ dag_run.conf.text }} resolves to the text I passed in CLI.
> {code}
> templated_command = """
> echo "ds = {{ ds }}"
> echo "prev_ds = {{ macros.datetime.strftime(prev_execution_date, "%Y-%m-%d") }}"
> echo "next_ds = {{ macros.datetime.strftime(next_execution_date, "%Y-%m-%d") }}"
> echo "text_through_conf = {{ dag_run.conf.text }}"
> """
> bash_operator = BashOperator(
> task_id='bash_task',
> bash_command=templated_command,
> dag=dag
> )
> {code}
> [1] [https://airflow.apache.org/cli.html#trigger_dag]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)