You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "stephenatgithub (via GitHub)" <gi...@apache.org> on 2024/03/08 02:41:30 UTC
[I] DAG did not run as scheduled [airflow]
stephenatgithub opened a new issue, #37987:
URL: https://github.com/apache/airflow/issues/37987
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.6.3
### What happened?
We have a DAG scheduled to run on 2024-03-08 08:00 HKT. However, it is not run as expected.
Checked linux log folder, 2024-03-08 folder is not found at /airflow/logs/dag_id=<second DAG>/run_id=scheduled__2024-03-08T00:00:00+00:00
We are using postgres and checked database table dag_run, the 2024-03-08 record is not found.
So there is no error message that we can google it.
This DAG run fine for a long time before and the last run in dashboard shows that yesterday, but not today.
This is the DAG code
```
from airflow.decorators import dag, task
from datetime import datetime
from airflow.timetables.trigger import CronTriggerTimetable
@dag(
start_date=datetime(2023,9,12),
schedule=CronTriggerTimetable("0 8 * * 1-5", timezone="Asia/Hong_Kong")
)
def xxx():
@task
def mssql_task():
from airflow.models import Variable
xxx_Variable = Variable.get("xxx", deserialize_json=True)
from airflow.providers.microsoft.mssql.hooks.mssql import MsSqlHook
mssql_hook = MsSqlHook(mssql_conn_id=xxx_Variable["mssql_conn_id"])
mssql_result = mssql_hook.get_records(sql=xxx_Variable["sql"])
mssql_task()
xxx()
```
for scheduler, we are using LocalExecutor with all setup
### What you think should happen instead?
We tried to check log file but no today folder is created.
We tried to check table dag_run, no today record is created.
We tried to search github issue history, but no error message can be looked at.
Wild guess...
It is not webserver problem.
It is not triggerer problem as we did not use it.
It is not DAG file problem as we did not change the code for 5 months.
It is not metadata DB problem as no record is created.
It is not parsing problem in scheduler as this DAG run as expected for long time.
It is not executing problem in scheduler as this DAG does not even run.
It may be scheduling problem in scheduler scheduler_job_runner.py
However, another DAG running at the same time on 2024-03-08 08:00 HKT worked fine. There is no relation between them.
### How to reproduce
1. create 2 separate different DAG running at the same time
2. the first and second DAG run as expected for a long time.
3. Suddenly, one of them did not run while another DAG keep running fine.
### Operating System
Ubuntu 22.04.2 LTS
### Versions of Apache Airflow Providers
Package Name Version Description
apache-airflow-providers-common-sql 1.5.2 Common SQL Provider<br>
apache-airflow-providers-ftp 3.4.2 File Transfer Protocol (FTP)<br>
apache-airflow-providers-http 4.4.2 Hypertext Transfer Protocol (HTTP)<br>
apache-airflow-providers-imap 3.2.2 Internet Message Access Protocol (IMAP)<br>
apache-airflow-providers-microsoft-mssql 3.4.1 Microsoft SQL Server (MSSQL)<br>
apache-airflow-providers-sftp 4.3.1 SSH File Transfer Protocol (SFTP)<br>
apache-airflow-providers-smtp 1.2.0 Simple Mail Transfer Protocol (SMTP)<br>
apache-airflow-providers-sqlite 3.4.2 SQLite<br>
apache-airflow-providers-ssh 3.7.1 Secure Shell (SSH)<br>
apache-airflow-providers-telegram 4.1.1 Telegram<br>
### Deployment
Virtualenv installation
### Deployment details
pip install "apache-airflow[statsd,ssh,smtp,http,ftp,sftp,common.sql,microsoft.mssql,telegram]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG did not run as scheduled [airflow]
Posted by "stephenatgithub (via GitHub)" <gi...@apache.org>.
stephenatgithub commented on issue #37987:
URL: https://github.com/apache/airflow/issues/37987#issuecomment-1989734279
I am not sure if I reported this issue here correctly ?
Is there anything I can work on to check the root cause with this problem.
Hopefully someone can shed some light.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG did not run as scheduled [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37987: DAG did not run as scheduled
URL: https://github.com/apache/airflow/issues/37987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG did not run as scheduled [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37987:
URL: https://github.com/apache/airflow/issues/37987#issuecomment-1984945941
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org