You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/23 13:52:51 UTC
[GitHub] [airflow] harrisjoseph opened a new issue, #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
harrisjoseph opened a new issue, #25905:
URL: https://github.com/apache/airflow/issues/25905
### Description
`airflow dags trigger` creates dag_runs with a specified `execution_date`. Since the deprecation of `execution_date`, some logic in the `infer_manual_data_interval()` method has been used to translate between `execution_date` and the new `logical_start_date` + `logical_end_date` fields. This logic sometimes produces incorrect dates.
We should add a new CLI & API endpoint + code path that accepts explicit `logical_start_date` + `logical_end_date` values, and provide a path that doesn't rely on the deprecated `execution_date`.
I'd like to find some time to do this myself and raise a PR, but can't guarantee that I'll get around to it in the next few months
### Use case/motivation
```
airflow dags trigger --logical_start_date {date} --logical_end_date {date
## produces a dag_run with those dates
```
Rather than the behaviour:
```
airflow dags trigger --executiond_date {date}
# Produces a dag_run with dates that are sometimes correct, sometimes incorrect, depending on whether "infer_manual_data_interval" likes them or not
```
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] harrisjoseph commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
harrisjoseph commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1225956990
Thank you for your response, I appreciate the input 🙂. I apologise for using the wrong terminology - I've been reading the code where logical_date is passed to `infer_manual_data_interval`, and haven't been following the conceptual shift from `execution_date` -> `logical_date` -> `data_interval` very closely.
Ultimately I'm trying to resolve the issue discussed [here](https://github.com/apache/airflow/discussions/25687) - a custom data interval isn't what I'm looking for - a _correct_ data interval is what I'd like to achieve:
```python
from airflow.timetables.interval import CronDataIntervalTimetable
import pytz
# pass 2022-02-14. I expect start_date = 2022-02-07 and end_date 2022-02-14
dt = datetime(2022,2,14, tzinfo=pytz.UTC)
tt = CronDataIntervalTimetable('0 0 * * 1', pytz.UTC)
tt.infer_manual_data_interval(run_after=dt)
# I get start_date 2022-01-31 and end_date 2022-02-07
DataInterval(start=DateTime(2022, 1, 31, 0, 0, 0, tzinfo=Timezone('UTC')), end=DateTime(2022, 2, 7, 0, 0, 0, tzinfo=Timezone('UTC')))
```
Perhaps the run_after date filter needs to be greater than inclusive rather than exclusive to make the behaviour match the previous behaviour of `airflow dags trigger -e`. Either way, I think we need to be able to pass something to `airflow dags trigger` that doesn't belong to the old forbidden world of execution_dates. Either we pass data_interval, or logical_date, or some different thing. But it would make sense for the resulting dag_run to have a data_interval which matches the input of the command as it was before the deprecation of `execution_date`, if possible?
Let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] harrisjoseph closed issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
harrisjoseph closed issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
URL: https://github.com/apache/airflow/issues/25905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1225998236
```python
# pass 2022-02-14. I expect start_date = 2022-02-07 and end_date 2022-02-14
# I get start_date 2022-01-31 and end_date 2022-02-07
```
I think this is simply a bug and there’s already an issue raised about it (not sure if a fix was proposed/merged yet), so perhaps we just need to make this as expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] harrisjoseph commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
harrisjoseph commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1226971338
I see no backport, or mention of the PR number between the two releases. I believe it has been pushed to 2.4.0 based on the new milestone on the PR.
https://github.com/apache/airflow/compare/2.3.3...2.3.4
https://github.com/apache/airflow/pull/22658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1224127896
Hopefully somene will pick it up. I think it's a useful one @uranusjr - any comments on that one ? Maybe some work is already happening there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1227594458
Oh I just recalled, it was dropped since backport was non-trivial (timetable changed too much between 2.3 and 2.4). Well 2.4.0 should be coming next month, so…
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] russellpierce commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by "russellpierce (via GitHub)" <gi...@apache.org>.
russellpierce commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1665285346
... an explicit start and end data interval would be very helpful. As of right now if you want to trigger a DAG run to start before end data interval there appears to be no way to do so... but early/partial evaluation of a data interval is a fairly valid thing to want to do - particularly in dev environments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1225070015
There’s no logical start/end date, data interval is the term. Being able to provide a custom interval does not make much sense with the current design since the interval is dictated by the timetable, and arbitrary values would make things inconsistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] harrisjoseph commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
harrisjoseph commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1226950641
Thanks. I had hoped that it had been fixed, but didn't see anything in 2.3.2 / 2.3.3 / 2.3.4. Also didn't find any issues which mentioned `infer_manual_data_interval`, so thought this hadn't been discussed at all.
I'll close this issue / discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25905: Create a replacement `airflow dags trigger` CLI / API endpoint which takes an explicit `logical_start_date`/`logical_end_date`
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25905:
URL: https://github.com/apache/airflow/issues/25905#issuecomment-1226963380
Hmm, the PR fixin this is marked as a part of 2.3.4 though. @ephraimbuddy Could you check if #22658 was backported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org