You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/09 20:51:12 UTC
[GitHub] [airflow] bluek1te opened a new issue, #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
bluek1te opened a new issue, #27582:
URL: https://github.com/apache/airflow/issues/27582
### Apache Airflow version
2.4.2
### What happened
New DAG with CronTriggerTimetable will sometimes miss the first run of the job. If I edit the schedule after the miss, it will usually recover and start scheduling on time. This doesn't seem to happen 100% of the time.
### What you think should happen instead
DAG implementing CronTriggerTimetable with a start_date occurring before the current time, with catchup disabled should run at the next time occurrence specified by the cron parameter.
As you can see here, the "next-run" field says that it is scheduled for 2022-11-09, 20:50:00, and then transforms to the next day without executing a run.
### How to reproduce
DAG Code Used:
```from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta
from airflow.timetables.trigger import CronTriggerTimetable
with DAG(
'cron-trigger-reproduce-bug',
description='cron-trigger-reproduce-bug',
start_date=datetime(2022, 1, 1),
catchup=False,
schedule=CronTriggerTimetable('50 14 * * *', timezone='America/Chicago'),
tags=['demo']
) as echo:
slack_hello = BashOperator(
task_id="echo",
bash_command="echo Hello World!"
)
```
I add the DAG, let Airflow pick it up, enable it in the web UI, and observe that the first run is missed.
If I modify the cron time again (for a later time), the DAG will recover.
### Operating System
Official Apache airflow 2.4.2 docker Image (alpine?)
### Versions of Apache Airflow Providers
Not relevant, example code doesn't use any providers.
### Deployment
Other Docker-based deployment
### Deployment details
Running Airflow in on prem k8s with 2 worker instances.
### Anything else
I am willing to try to write up a PR for this with some mentoring/pointers on where to look, thanks!
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1310661445
ACK, I'm going to do some more sandboxing in my environment to see if I can get a deterministic set of steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1309360233
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mai-nakagawa commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
mai-nakagawa commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311128113
@bluek1te Do you see anything strange in Airflow Scheduler log? This could happen if Airflow Scheduler is down when it is supposed to trigger a DAG run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1312542749
Tested this locally and it worked fine on local airflow setup with docker compose, so am going back to looking at config/network issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mai-nakagawa commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
mai-nakagawa commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311100780
I'm also trying to reproduce the reported issue. However, I haven't seen the issue so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1315550652
Hey all - there was an issue with our Airflow configuration that caused the DAG's to reload in the order of milliseconds. This ended up exacerbating the issue which I believe is being discussed here: https://github.com/apache/airflow/issues/27399
The occurrence where it so happened to be the "first" run that missed was by chance and we ended up observing much more than the first being missed. Will close this issue and follow #27399, thanks!.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1310068044
Just found #27399 which looks very similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1309968918
First we need to debug this and get the steps for reproduction more deterministic. For example, are there patterns you can notice when the scheduler is able to “recover”? Perhaps when you change the schedule to some specific values? When you do that in some certain points of time?
Also cc @mai-nakagawa for ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te closed issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te closed issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
URL: https://github.com/apache/airflow/issues/27582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311861535
The dag ID is present in the `dags` table though for these specific dags:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311185737
Sure, just tested it again, before 9:25PM CST:
After 9:25PM CST:
Logs from scheduler (Debug turned on, let me know if you want logs without it turned on:
At 03:24:59
```
[2022-11-11 03:24:59,400] {retries.py:84} DEBUG - Running SchedulerJob._create_dagruns_for_dags with retries. Try 1 of 3
[2022-11-11 03:24:59,406] {retries.py:84} DEBUG - Running SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2022-11-11 03:24:59,411] {retries.py:84} DEBUG - Running SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2022-11-11 03:24:59,412] {retries.py:84} DEBUG - Running SchedulerJob._schedule_all_dag_runs with retries. Try 1 of 3
[2022-11-11 03:24:59,420] {scheduler_job.py:342} DEBUG - No tasks to consider for execution.
[2022-11-11 03:24:59,421] {base_executor.py:163} DEBUG - 0 running task instances
[2022-11-11 03:24:59,421] {base_executor.py:164} DEBUG - 0 in queue
[2022-11-11 03:24:59,421] {base_executor.py:165} DEBUG - 32 open slots
[2022-11-11 03:24:59,421] {base_executor.py:174} DEBUG - Calling the <class 'airflow.executors.celery_executor.CeleryExecutor'> sync method
[2022-11-11 03:24:59,421] {celery_executor.py:330} DEBUG - No task to query celery, skipping sync
[2022-11-11 03:24:59,421] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,421] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,421] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,421] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,422] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:24:59,432] {scheduler_job.py:879} DEBUG - Next timed event is in 0.864662
[2022-11-11 03:24:59,432] {scheduler_job.py:881} DEBUG - Ran scheduling loop in 0.03 seconds
```
At 03:25:00
```
[2022-11-11 03:25:00,298] {retries.py:84} DEBUG - Running SchedulerJob._create_dagruns_for_dags with retries. Try 1 of 3
[2022-11-11 03:25:00,303] {retries.py:84} DEBUG - Running SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2022-11-11 03:25:00,309] {retries.py:84} DEBUG - Running SchedulerJob._get_next_dagruns_to_examine with retries. Try 1 of 3
[2022-11-11 03:25:00,309] {retries.py:84} DEBUG - Running SchedulerJob._schedule_all_dag_runs with retries. Try 1 of 3
[2022-11-11 03:25:00,317] {scheduler_job.py:342} DEBUG - No tasks to consider for execution.
[2022-11-11 03:25:00,318] {base_executor.py:163} DEBUG - 0 running task instances
[2022-11-11 03:25:00,318] {base_executor.py:164} DEBUG - 0 in queue
[2022-11-11 03:25:00,318] {base_executor.py:165} DEBUG - 32 open slots
[2022-11-11 03:25:00,318] {base_executor.py:174} DEBUG - Calling the <class 'airflow.executors.celery_executor.CeleryExecutor'> sync method
[2022-11-11 03:25:00,318] {celery_executor.py:330} DEBUG - No task to query celery, skipping sync
[2022-11-11 03:25:00,318] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,318] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,318] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,318] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,318] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,319] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,319] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,319] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,319] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,319] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,319] {manager.py:275} DEBUG - Received message of type DagParsingStat
[2022-11-11 03:25:00,331] {scheduler_job.py:879} DEBUG - Next timed event is in 4.446577
[2022-11-11 03:25:00,331] {scheduler_job.py:881} DEBUG - Ran scheduling loop in 0.03 seconds
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311026917
Sandboxed this all day today. It looks like every time that I have tested this so far, I have not been able to get the first run to occur
Step 1: Create new DAG with ID that is a completely new entry in the database, with a CronTriggerTimetable that is scheduled to occur 5 minutes later than my current time. (I have added a timezone attribute and am setting the cron time to my expected local time)
Step 2: Enable the DAG in the WebUI (Before the DAG is expected to trigger)
Step 3: Wait for the time the DAG should execute
Step 4: Observe that the DAG has not executed.
It seems to me like this would be a pretty big issue that more people would run into, so it could still very well be a configuration error on my part. Would anyone else be able to replicate these steps and let me know if the same issue persists for them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mai-nakagawa commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
mai-nakagawa commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311252990
Hmmm, there are only debug logs. I found nothing strange.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1311855348
I peeked into the `jobs` table for the database. It looks like these jobs are getting created, but they don't have the correct (or any) dag ID associated with them. My other jobs that don't use timetable seem to be working fine though (and have Dag ID's). Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bluek1te commented on issue #27582: CronTriggerTimetable doesn't run the first scheduled job after creating a new DAG.
Posted by GitBox <gi...@apache.org>.
bluek1te commented on issue #27582:
URL: https://github.com/apache/airflow/issues/27582#issuecomment-1312198837
The above null dag ID entries were just from me restarting my scheduler and triggerer pods manually, so not really anything interesting. Am going to try running this code on my local airflow today and seeing if I still get this issue. (Wiping the database did not do anything)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org