You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/12 22:29:28 UTC

[GitHub] [airflow] alexbegg opened a new issue, #23689: Data Interval Start off when manually triggering with a specific local date

alexbegg opened a new issue, #23689:
URL: https://github.com/apache/airflow/issues/23689

   ### Apache Airflow version
   
   2.2.5
   
   ### What happened
   
   When I use the date picker in the “Trigger DAG w/ config” page to choose a specific logical date for some reason on a daily DAG the Data Interval Start (circled in red) )is 2 days before the logical date (circled in blue), instead of the same as the logical date. And the Data Interval End is one day before the logical date. So the interval is the correct length, but on wrong days.
   
   ![Screen Shot 2022-05-11 at 5 14 10 PM](https://user-images.githubusercontent.com/45696489/168159891-b080273b-4b22-4ef8-a2ae-98327a503f9f.png)
   
   I encountered this with a DAG with a daily schedule which typically runs at 09:30 UTC. I am testing this in a dev environment (with catchup off) and trying to trigger a run for 2022-05-09 09:30:00. I would expect the data interval to start at that same time and the data interval end to be 1 day after.
   
   It has nothing to do with the previous run since that was way back on 2022-04-26
   
   ### What you think should happen instead
   
   The data interval start date should be the same as the data interval start date
   
   ### How to reproduce
   
   I made a sample DAG as shown below:
   ```python
   import pendulum
   from airflow.models import DAG
   from airflow.operators.python import PythonOperator
   
   
   def sample(data_interval_start, data_interval_end):
       return "data_interval_start: {}, data_interval_end: {}".format(str(data_interval_start), str(data_interval_end))
   
   
   args = {
       'start_date': pendulum.datetime(2022, 3, 10, 9, 30)
   }
   with DAG(
       dag_id='sample_data_interval_issue',
       default_args=args,
       schedule_interval='30 9 * * *'  # 09:30 UTC
   ) as sample_data_interval_issue:
       task = PythonOperator(
           task_id='sample',
           python_callable=sample
       )
   
   ```
   
   I then start it to start a scheduled DAG run (`2022-05-11, 09:30:00 UTC`), and the `data_interval_start` is the same as I expect, `2022-05-11T09:30:00+00:00`.
   
   However, when I went to "Trigger DAG w/ config" page and in the date chooser choose `2022-05-09 09:30:00+00:00`, and then triggered that. It shows the run datetime is `2022-05-09, 09:30:00 UTC`, but the `data_interval_start` is incorrectly set to `2022-05-08T09:30:00+00:00`, 2 days before the date I choose.
   
   ### Operating System
   
   Debian GNU/Linux 11 (bullseye)
   
   ### Versions of Apache Airflow Providers
   
   N/A
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb closed issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
ashb closed issue #23689: Data Interval wrong when manually triggering with a specific logical date
URL: https://github.com/apache/airflow/issues/23689


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] alexbegg commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
alexbegg commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1147705383

   I have a question, that is sort of on this topic. Is it possible to override a Data Interval when triggering a DAG run manually? For example if my DAG has a monthly schedule, could I change the Data Interval for a single manual run to be 15 days instead, mabe using Params?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] alexbegg commented on issue #23689: Data Interval Start wrong when manually triggering with a specific local date

Posted by GitBox <gi...@apache.org>.
alexbegg commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1125487336

   I confirmed, if I triggered the DAG without using "Trigger DAG w/ config", it had the datetime of "now for a logical date, and the data interval end is the end of the most recent interval (09:30 UTC today), along the lines of what you said.
   
   However if I explicitly choose the logical date, I would expect it to either do the same as that, at the very least.
   
   On a related note, I sort of think it should be possible to choose if the interval is before or after the logical date when you triggering a DAg with a custom logical date. But that can be brought up as a separate request if needed.
   
   The issue here is the data interval is not working as you are describing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] seongwon77 commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
seongwon77 commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1176860591

   I experienced the same situation.   When I set the logical date to `2022-07-03T00:00:00` for a daily job and manually triggers it, it runs with the data interval as `2022-07-01T00:00:00` ~ `2022-07-02T00:00:00`.   I expected to run for `2022-07-02T00:00:00` ~ `2022-07-03T00:00:00` though.
   
   Is this a valid behavior?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] park-peter commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
park-peter commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1201552428

   But it still behaves differently than with the scheduled runs. Below are with `schedule_interval='@daily'`
   
   For scheduled runs:
   logical date = `2022-08-01T00:00:00`
   start_date = `2022-08-01T00:00:00`
   end_date = `2022-08-02T00:00:00`
   
   For manual runs, intentionally triggering it not exactly on the trigger date:
   logical date = `2022-08-01T09:30:00`
   start_date = `2022-07-31T00:00:00`
   end_date = `2022-08-01T00:00:00`
   
   For manual runs, intentionally triggering on the trigger date:
   logical date = `2022-08-01T00:00:00`
   start_date = `2022-07-30T00:00:00`
   end_date = `2022-07-31T00:00:00`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] park-peter commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
park-peter commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1201479935

   I ran into the same situation as referenced in #25408. @seongwon77 even though the UI shows the data interval with 2 schedules off, the actual date that the dag run processes is a correct one, i.e. if you manually run a dag for `2022-07-03T00:00:00`, `{{ ds_nodate }}` will be `20220703`. The `prev_ds` and `next_ds` will also be `2022-07-03` as they are equal when a run is manually triggered. It's just the UI that is off.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] park-peter commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
park-peter commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1201541011

   @uranusjr I take what I said back about the issue only existing in the UI. 
   ![image](https://user-images.githubusercontent.com/46539900/182213651-145102bf-e3aa-4eb0-94a5-281663acfe3a.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1146687380

   Do we think we can solve it somehow @uranusjr ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1201519730

   @park-peter Just to make sure, what do `{{ data_interval_start }}` and `{{ data_interval_end }}` point to in the run?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #23689: Data Interval Start wrong when manually triggering with a specific local date

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1125481834

   By default, manually triggered runs have their logical date at the _end_ of the data interval, not the start. That still does not explain the difference though (I would expect the data interval to be between May 8–9).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1201544452

   Looks like this is an issue in the timetable implementation when the logical date falls exactly on the trigger date.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #23689: Data Interval wrong when manually triggering with a specific logical date

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23689:
URL: https://github.com/apache/airflow/issues/23689#issuecomment-1201958599

   The `@daily` schedule interval is specifically dfefined as _create a run for each calendar day_, so the data interval must be between two midnights. The logical date is derived from the data interval, and for a scheduled date it is natural to use one of the terminal times as the logical date (the fact Airflow uses the _start_ is a historical issue; it should have used the end, but it is what it is and we cannot change it now).
   
   Therefore, for a scheduled run,
   
   ```
   logical date = 2022-08-01T00:00:00
   data_interval_start = 2022-08-01T00:00:00
   data_interval_end = 2022-08-02T00:00:00
   ```
   
   For a manual run, since for another historical issue (the logical date used to need to be _unique_), the manual trigger time is used as the logical date. But the data interval inferred from the logical date still needs to conform to the original definition, so it is “aligned” to the nearest interval that makes sense. Therefore
   
   ```
   logical date = 2022-08-01T09:30:00
   data_interval_start = 2022-07-31T00:00:00
   data_interval_end = 2022-08-01T00:00:00
   ```
   
   The only thing that does not work “correctly”, given the above rules, is when you specify a logical date that falls directly on the data interval’s end. The correct behaviour should be
   
   ```
   logical date = 2022-08-01T00:00:00
   start_date = 2022-07-31T00:00:00
   end_date = 2022-08-01T00:00:00
   ```
   
   because this is the cloest interval that the manual run can fit in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org