You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:44:03 UTC
[GitHub] [airflow] hkarakaki opened a new issue #16389: SLAs for only one task in a DAG
hkarakaki opened a new issue #16389:
URL: https://github.com/apache/airflow/issues/16389
**Apache Airflow version**: 2.1.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): NA
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Ubuntu 18.04.5
- **Kernel** (e.g. `uname -a`): Linux ip-172-20-1-237 5.4.0-1049-aws #51~18.04.1-Ubuntu SMP Fri May 14 18:38:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**:
- **Others**:
**What happened**:
We migrated from 1.10.15 this week, and noticed SLA misses are not being reported in UI and callback isn't triggered. Usually we have some checkpoint task in our largest DAGs and the SLA is defined only for that task, the task also helps us set up cross-DAG dependencies.
**What you expected to happen**:
SLA misses are registered in UI and trigger the callback.
**What do you think went wrong?**
I think tasks without SLA should be excluded from the loop here: https://github.com/apache/airflow/blob/304e174674ff6921cb7ed79c0158949b50eff8fe/airflow/jobs/scheduler_job.py#L411-L438
**How to reproduce it**:
With this DAG you can check that the SLA is not detected.
```Python
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
with DAG(
dag_id="sla_trigger",
schedule_interval="*/5 * * * *",
start_date=datetime(2021, 6, 10),
) as dag:
BashOperator(
task_id="bash_task",
bash_command="sleep 30",
sla=timedelta(seconds=2),
)
BashOperator(
task_id="bash_task_2",
bash_command="sleep 30",
)
```
But to get the error mentioned below, you need to add a callback function.
**Anything else we need to know**:
This error happens every time for every DAG that has SLA on some tasks, and the same DAG definition worked fine in v1.10.15.
<details><summary>Scheduler's log</summary>
We've seen this error in the scheduler for all the DAGs:<br>
[2021-06-09 20:30:57,679] {scheduler_job.py:396} INFO - Running SLA Checks for company_dag<br>
[2021-06-09 20:30:57,763] {scheduler_job.py:569} ERROR - Error executing SlaCallbackRequest callback for file: /opt/airflow/tasks/airflow_dags/eu-central-1/company_dag.py<br>
Traceback (most recent call last):<br>
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 565, in execute_callbacks<br>
self.manage_slas(dagbag.dags.get(request.dag_id)) <br>
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/session.py", line 70, in wrapper<br>
return func(*args, session=session, **kwargs) <br>
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/scheduler_job.py", line 433, in manage_slas<br>
if following_schedule + task.sla < timezone.utcnow():<br>
TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'NoneType'<br>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tseruga commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
tseruga commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-906537920
To add some more findings from my personal testing - this seems to only appear when using the context manager style of defining a DAG (e.g. `with DAG(...) as dag:`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Tonkonozhenko commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
Tonkonozhenko commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-874621123
@jhtimmins any updated? We faced the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #16389:
URL: https://github.com/apache/airflow/issues/16389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tazimmerman commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
tazimmerman commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-939135249
It appears that commit [2fd3f27e](https://github.com/apache/airflow/commit/2fd3f27ebd2a4f695968fb7cc5d119e087820928#diff-3101df8e7c54fe1b9305774befb54b31edfe84cb81425a300e70142bc07637d4R408) will resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tazimmerman commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
tazimmerman commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-939127691
>
>
> To add some more findings from my personal testing - this seems to only appear when using the context manager style of defining a DAG (e.g. `with DAG(...) as dag:`)
As of 2.1.4 this also happens without the context manager.
```python
Traceback (most recent call last):
File "/local/airflow/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 545, in execute_callbacks
elif isinstance(request, SlaCallbackRequest):
File "/local/airflow/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/local/airflow/lib/python3.8/site-packages/airflow/dag_processing/processor.py", line 413, in manage_slas
if following_schedule + task.sla < timezone.utcnow():
TypeError: unsupported operand type(s) for +: 'datetime.datetime' and 'NoneType'
```
From what I can tell it's because the check at the start of the `manage_slas` function (in `airflow/dag_processing/processor.py`) will short-circuit when **no** tasks have a valid SLA, but the rest of the code assumes that all tasks will have a valid SLA (by _valid_ I mean it's a `timedelta` and thus comparable).
```python
while dttm < timezone.utcnow():
following_schedule = dag.following_schedule(dttm)
if following_schedule + task.sla < timezone.utcnow():
session.merge(
SlaMiss(task_id=ti.task_id, dag_id=ti.dag_id, execution_date=dttm, timestamp=ts)
)
dttm = dag.following_schedule(dttm)
```
I'm not sure if there are other repercussions but it seems like an easy fix is to change that check line to:
```python
if task.sla and following_schedule + task.sla < timezone.utcnow():
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ReadytoRocc commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
ReadytoRocc commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-1047972705
I have tested using [astronomer ap-airflow](https://quay.io/repository/astronomer/ap-airflow) image `quay.io/astronomer/ap-airflow:2.2.0-onbuild`, and no longer see this behavior. SLA misses are now appearing in the UI for `bash_task` using the provided DAG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-859585028
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tazimmerman edited a comment on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
tazimmerman edited a comment on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-939135249
It appears that commit https://github.com/apache/airflow/commit/2fd3f27ebd2a4f695968fb7cc5d119e087820928 will resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tazimmerman edited a comment on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
tazimmerman edited a comment on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-939135249
It appears that commit https://github.com/apache/airflow/commit/2fd3f27ebd2a4f695968fb7cc5d119e087820928#diff-3101df8e7c54fe1b9305774befb54b31edfe84cb81425a300e70142bc07637d4R408 will resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Tonkonozhenko commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
Tonkonozhenko commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-874621123
@jhtimmins any updated? We faced the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #16389: SLAs for only one task in a DAG
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #16389:
URL: https://github.com/apache/airflow/issues/16389#issuecomment-1047982861
great. closing the issue then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org