You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "CYarros10 (via GitHub)" <gi...@apache.org> on 2023/07/07 16:34:45 UTC
[GitHub] [airflow] CYarros10 opened a new issue, #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true
CYarros10 opened a new issue, #32429:
URL: https://github.com/apache/airflow/issues/32429
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Airflow 2.4.3
I created a DAG with:
```
default_args = {
"owner": "me",
"depends_on_past": False,
"email": [""],
"email_on_failure": False,
"email_on_retry": False,
"retries": 1,
"retry_delay": timedelta(minutes=2),
"start_date": datetime(2023, 7, 4),
"sla": timedelta(minutes=25),
}
with models.DAG(
f"the_new_reporting_dag_{VERSION}",
description=f"Export metadata to BigQuery location and create reporting table.",
schedule_interval="0 0 * * *", # daily
tags=tags,
default_args=default_args,
user_defined_macros=user_defined_macros,
is_paused_upon_creation=True,
catchup=True,
max_active_runs=1,
dagrun_timeout=timedelta(minutes=30),
) as dag:
```
and ran it on 7/7/2023.
This started three dag runs with logical dates of 7/4/2023, 7/5/2023, 7/6/2023.
An SLA Miss entry SHOULD have these logical dates. Yet it does not. For whatever reason, SLA Miss entries were for 2023-07-06 00:00:00 and 2023-07-07 00:00:00, with SLA timestamps of 2023-07-07, 16:10:54 and 2023-07-07, 16:12:04 respectively.
Why? SLA Miss table seems to have incorrectly inserted the wrong logical dates AND completely missed the third DAG run.
### What you think should happen instead
SLA Miss entries should have occurred for the correct logical dates of:
2023-07-04 00:00:00
2023-07-05 00:00:00
2023-07-06 00:00:00
Because they all missed their 25 minute SLA (since they were all ran on 2023-07-07)
### How to reproduce
1. Create a DAG with an SLA of 20 minutes, start date of 3 days ago, and catchup = true.
### Operating System
cloud composer on GCP
### Versions of Apache Airflow Providers
_No response_
### Deployment
Google Cloud Composer
### Deployment details
Airflow 2.4.3, Cloud Composer 2.1.8
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] CYarros10 commented on issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true
Posted by "CYarros10 (via GitHub)" <gi...@apache.org>.
CYarros10 commented on issue #32429:
URL: https://github.com/apache/airflow/issues/32429#issuecomment-1625694834
I'm beginning to think that SLA Miss Logical Date != DAG Run Logical Date.
SLA Miss Logical Date == DAG Run Data Interval End Date.
that is incredibly confusing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #32429:
URL: https://github.com/apache/airflow/issues/32429#issuecomment-1634041944
@CYarros10 Given what we know about the nature of the SLA feature and the current efforts to try and change how it works, I don't think there's much that can be done on this issue so I am tempted to close it. I would encourage you however to speak with those involved on the AIP above to see if your needs can be met in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #32429:
URL: https://github.com/apache/airflow/issues/32429#issuecomment-1625700411
FYI. SLA feature is broken and we know it. There is an effort to redefine how it works https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-57+SLA+as+a+DAG-Level+Feature where you can find what's broken for it - and it's not very likely there will be any fix for what you observe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #32429: SLA Logical Date is incorrect, SLA Miss Table missing entries for DAGs with catchup=true
URL: https://github.com/apache/airflow/issues/32429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org