You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "luederm (via GitHub)" <gi...@apache.org> on 2023/04/14 19:59:24 UTC
[GitHub] [airflow] luederm opened a new issue, #30647: Scheduler crashes w/ divide by 0 error when expanding with empty list
luederm opened a new issue, #30647:
URL: https://github.com/apache/airflow/issues/30647
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
I have a task which returns a list of items. Sometimes this list can be empty. A downstream taskgroup is expanded with this list. When the list is empty, the scheduler crashes with a ZeroDivisionError.
This is in version 2.5.1.
```
Traceback (most recent call last):
File "/data/srv/airflow_builds/43ad18c873a6c1dab54c69896feb242823302142/dags/pipeline_input_to_s3.py", line 136, in <module>
dag.test()
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/dag.py", line 2533, in test
schedulable_tis, _ = dr.update_state(session=session)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
return func(*args, **kwargs)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/dagrun.py", line 563, in update_state
info = self.task_instance_scheduling_decisions(session)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
return func(*args, **kwargs)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/dagrun.py", line 710, in task_instance_scheduling_decisions
schedulable_tis, changed_tis, expansion_happened = self._get_ready_tis(
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/dagrun.py", line 793, in _get_ready_tis
if not schedulable.are_dependencies_met(session=session, dep_context=dep_context):
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
return func(*args, **kwargs)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1070, in are_dependencies_met
for dep_status in self.get_failed_dep_statuses(dep_context=dep_context, session=session):
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1091, in get_failed_dep_statuses
for dep_status in dep.get_dep_statuses(self, session, dep_context):
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 107, in get_dep_statuses
yield from self._get_dep_statuses(ti, session, cxt)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/ti_deps/deps/trigger_rule_dep.py", line 93, in _get_dep_statuses
yield from self._evaluate_trigger_rule(ti=ti, dep_context=dep_context, session=session)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/ti_deps/deps/trigger_rule_dep.py", line 219, in _evaluate_trigger_rule
.filter(or_(*_iter_upstream_conditions()))
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/ti_deps/deps/trigger_rule_dep.py", line 191, in _iter_upstream_conditions
map_indexes = _get_relevant_upstream_map_indexes(upstream_id)
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/ti_deps/deps/trigger_rule_dep.py", line 138, in _get_relevant_upstream_map_indexes
return ti.get_relevant_upstream_map_indexes(
File "/data/srv/pyenv/versions/3.9.9/envs/43ad18c873a6c1dab54c69896feb242823302142/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2652, in get_relevant_upstream_map_indexes
ancestor_map_index = self.map_index * ancestor_ti_count // ti_count
ZeroDivisionError: integer division or modulo by zero
```
### What you think should happen instead
Downstream tasks should be skipped.
### How to reproduce
Expand a task group with an empty list.
### Operating System
Mac+linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala closed issue #30647: Scheduler crashes w/ divide by 0 error when expanding with empty list
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #30647: Scheduler crashes w/ divide by 0 error when expanding with empty list
URL: https://github.com/apache/airflow/issues/30647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #30647: Scheduler crashes w/ divide by 0 error when expanding with empty list
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #30647:
URL: https://github.com/apache/airflow/issues/30647#issuecomment-1509162300
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #30647: Scheduler crashes w/ divide by 0 error when expanding with empty list
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #30647:
URL: https://github.com/apache/airflow/issues/30647#issuecomment-1509414330
Fixed by #30084 in 2.5.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org