You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/13 10:38:29 UTC
[GitHub] [airflow] awildturtok opened a new issue #20850: Dag using schedule_interval is not scheduled
awildturtok opened a new issue #20850:
URL: https://github.com/apache/airflow/issues/20850
### Apache Airflow version
2.1.3
### What happened
I have a DAGs schedule_interval set to `40 9 * * 4` so i expect it to run at Thursday 9:40. This however has not happened. I've read the docs and they state, the scheduler will try to schedule for the end of the period, which also seems to not happen (I'd assume this to mean `40 9 * * 3` would run at Thursday 9:40)?
From the scheduler logs:
```
INFO - Setting next_dagrun for Am300_Pipeline to 2022-01-13 09:40:00+00:00
```
### What you expected to happen
My DAG get's queued every Thursday at 9:40.
### How to reproduce
```python
with DAG(dag_id="Pipeline",
schedule_interval="40 9 * * 4",
start_date=datetime(2020, 1, 1),
catchup=False,
default_args={
"email_on_failure": True,
"email": email
}
) as dag:
```
### Operating System
Dockerfile based on `apache/airflow:2.1.3-python3.8`
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
Airflow is run using podman inside a container with the following script (I know it's suboptimal, working on decomposing it), configured to use a LocalExecutor and communicating using a postgres inside the (podman) pod.
```
# start webserver
airflow webserver &>> ./logs/current.log &
# start the scheduler
airflow scheduler &>> ./logs/current.log &
# tail logs for container logs
tail -f ./logs/current.log
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] awildturtok commented on issue #20850: Dag using schedule_interval is not scheduled
Posted by GitBox <gi...@apache.org>.
awildturtok commented on issue #20850:
URL: https://github.com/apache/airflow/issues/20850#issuecomment-1014509508
> This can cause confusion, especially when you also run your dag manually. Check out the logs of the runs to see when your dag ran.
Manual dag runs interfere with the schedule of scheduled dag-runs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #20850: Dag using schedule_interval is not scheduled
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #20850:
URL: https://github.com/apache/airflow/issues/20850#issuecomment-1042631325
I'm not able to reproduce.
Converting to discussion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chla0089 commented on issue #20850: Dag using schedule_interval is not scheduled
Posted by GitBox <gi...@apache.org>.
chla0089 commented on issue #20850:
URL: https://github.com/apache/airflow/issues/20850#issuecomment-1014287705
Hi,
are you aware the the timestamps displayed show the start of the current interval?
E.g. if it shows now (on 2021-01-17) that the next dag run is `2022-01-13 09:40:00+00:00`, it means your dag will be executed at the end of the interval which is at `2022-01-20 09:40:00+00:00`.
In the list of runs it will also display the scheduled runs only by the start of the interval and not the actual time of the dag execution. This can cause confusion, especially when you also run your dag manually. Check out the logs of the runs to see when your dag ran.
Maybe this helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chla0089 commented on issue #20850: Dag using schedule_interval is not scheduled
Posted by GitBox <gi...@apache.org>.
chla0089 commented on issue #20850:
URL: https://github.com/apache/airflow/issues/20850#issuecomment-1014561319
>
> Manual dag runs interfere with the schedule of scheduled dag-runs?
No, it does not interfere with the runs itself, but with the order they are displayed on the airflow-webserver when you click on the dag. The displayed time is always the start of the interval (not the end at which the dag is executed).
Example: Assuming you have a dag that runs daily at 8am (`0 8 * * *`) and you execute the same dag manually at 7.45am. If you check the dag at 9am today (`2022-01-17`), one assumes when looking at the tree that you see execution times at `2022-01-16 08:00` (scheduled), `2022-01-17 07:45` (manually), and `2022-01-17 08:00` (scheduled). However, that is not the case, even though the dag was exactly at those times executed. The displayed times of those three executions are: `2022-01-15 08:00` (scheduled), `2022-01-16 08:00` (scheduled), and `2022-01-17 07:45` (manually), because the intervals started at these times.
Ergo, when you check if your dag was successfully executed at `22-01-13 09:40:00+00:00`, you have to look for the timestamp `2022-01-06 09:40:00+00:00`.
When the next dag run states `22-01-13 09:40:00+00:00`, the dag will be executed at the end of the 7-day interval that started at the given timestamp.
It is also possible that there is another bug or error. I had a similar issue than you and was looking for the reason only to find out that I interpreted the timestamps wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org