[GitHub] [airflow] MatrixManAtYrService opened a new issue #20557: Scheduled runs use user-supplied default dagrun parameters.

[GitBox <gi...@apache.org>]

MatrixManAtYrService opened a new issue #20557:
URL: https://github.com/apache/airflow/issues/20557


   ### Description
   
   If you unpause this DAG, it will immediately create a failed dagrun:
   
   ```
   from datetime import datetime, timedelta
   from textwrap import dedent
   
   from airflow import DAG
   from airflow.models.param import Param
   from airflow.operators.python_operator import PythonOperator
   
   thirty_years = 1577862000  # seconds
   
   with DAG(
       "params_int_fivetoten",
       start_date=datetime(1970, 1, 1),
       schedule_interval=timedelta(seconds=thirty_years),
       catchup=True,
       params={"x": Param(7, type="integer", minimum=5, maximum=10)},
       render_template_as_native_obj=True,
       doc_md=dedent(
           """
           # Purpose
   
           Check if scheduled runs receive default dagrun params.
   
           ## Steps
   
           1. Unpause
           2. Wait for dag run to complete
   
           ## Expectations
   
           - All tasks succeed
           """
       ),
   ) as use_defaults:
   
       def fail_if_invalid(x):
           assert 5 < x < 10
   
       PythonOperator(
           task_id="reference_param",
           python_callable=fail_if_invalid,
           op_args=["{{ dag_run.conf['x'] }}"],
       )
   ```
   
   This happens because the default dagrun params are only provided to manually triggered runs (and only if you explicitly run with params).  If the scheduler creates a dagrun, or if the user runs without thinking about params, `{}` is supplied and `"x"` is not found in in it.
   
   I would like to request that the default params be used...
    - when dagruns are manually triggered without special thought to parameters
    - and when the scheduler creates a dagrun
   
   ### Use case/motivation
   
   I have a DAG that tests an OCI image and pushes test results to a dashboard.  Most of the time, I want to test the image tagged `latest`. But sometimes I want to investigate the environment created by testing a tag other than `latest`.
   
   Ideally, I could just specify default parameters:
   ```
   {
       "use_tag": "latest"
   }
   ```
   ...and then ignore them most of the time.  I would change them from their defaults only on the occasions where I need to recreate a bug at an old version.
   
   Instead I'm working around this by having tasks that check to see if there's anything in `dag_run.conf` and honors them if present.  Otherwise it uses hard-coded defaults.  That is, I'm implementing defaults in task code, when there's already an Airflow-native default parameter mechanism--it's just not compatible with scheduled runs.
   
   ### Related issues
   
   https://github.com/apache/airflow/issues/12817
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] MatrixManAtYrService commented on issue #20557: Make scheduled runs use user-supplied default dagrun parameters

[GitBox <gi...@apache.org>]

MatrixManAtYrService commented on issue #20557:
URL: https://github.com/apache/airflow/issues/20557#issuecomment-1002755055


   Oh man, I'm silly.  This is already how it works.  I was just referencing `conf` and not `params`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] MatrixManAtYrService closed issue #20557: Make scheduled runs use user-supplied default dagrun parameters

[GitBox <gi...@apache.org>]

MatrixManAtYrService closed issue #20557:
URL: https://github.com/apache/airflow/issues/20557


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org