You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/11 10:22:26 UTC
[GitHub] [airflow] uranusjr opened a new pull request, #22904: Call mapped_dependants only on the original task
uranusjr opened a new pull request, #22904:
URL: https://github.com/apache/airflow/pull/22904
We've made change on this in the scheduler, but need to match it in the BackfillJob.
Should be the last piece to fix #22626.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095447937
Yeah I just added those two to the test DAG (_without the proposed fix) and CI still passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1094981510
```diff
diff --git a/tests/dags/test_mapped_classic.py b/tests/dags/test_mapped_classic.py
index 3880cc74f..4c0b4b1e7 100644
--- a/tests/dags/test_mapped_classic.py
+++ b/tests/dags/test_mapped_classic.py
@@ -32,3 +32,4 @@ def consumer(value):
with DAG(dag_id='test_mapped_classic', start_date=days_ago(2)) as dag:
PythonOperator.partial(task_id='consumer', python_callable=consumer).expand(op_args=make_arg_lists())
+ PythonOperator.partial(task_id='consumer', python_callable=consumer).expand(op_args=[1,2,3])
diff --git a/tests/dags/test_mapped_taskflow.py b/tests/dags/test_mapped_taskflow.py
index 34f6ae3d7..31c132f7c 100644
--- a/tests/dags/test_mapped_taskflow.py
+++ b/tests/dags/test_mapped_taskflow.py
@@ -29,3 +29,4 @@ with DAG(dag_id='test_mapped_taskflow', start_date=days_ago(2)) as dag:
print(repr(value))
consumer.expand(value=make_list())
+ consumer.expand(value=[1,2,3])
```
Should do it I think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1094959233
We should be able to add a test for this in test_backfilljob somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095457156
Oh. https://github.com/apache/airflow/blob/main/scripts/docker/entrypoint_ci.sh#L353-L357
No. 😱
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095490525
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095358722
Hmm that looks weird. The bug is when an upstream task finishes and wants to scan its downstream tasks; so whether the upstream is mapped from a literal or not should not matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095455487
Hmmmm… does the CI run long-running tests by default…?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095450527
```
FAILED tests/jobs/test_backfill_job.py::TestBackfillJob::test_mapped_dag[test_mapped_classic-DebugExecutor] - RuntimeError: Cannot check for mapped_dependants when not attached to a DAG
========================================================================================================================== 1 failed, 1 passed, 46 deselected, 18 warnings in 19.09s ==========================================================================================================================
~/code/airflow/airflow fix-mapping-removed-tasks-bug* 20s
airflow ❯ git di [20:07] [1] !10947
diff --git a/tests/dags/test_mapped_classic.py b/tests/dags/test_mapped_classic.py
index 3880cc74f..9759b4c39 100644
--- a/tests/dags/test_mapped_classic.py
+++ b/tests/dags/test_mapped_classic.py
@@ -32,3 +32,4 @@ def consumer(value):
with DAG(dag_id='test_mapped_classic', start_date=days_ago(2)) as dag:
PythonOperator.partial(task_id='consumer', python_callable=consumer).expand(op_args=make_arg_lists())
+ PythonOperator.partial(task_id='consumer_literal', python_callable=consumer).expand(op_args=[[1],[2],[3]])
diff --git a/tests/dags/test_mapped_taskflow.py b/tests/dags/test_mapped_taskflow.py
index 34f6ae3d7..31c132f7c 100644
--- a/tests/dags/test_mapped_taskflow.py
+++ b/tests/dags/test_mapped_taskflow.py
@@ -29,3 +29,4 @@ with DAG(dag_id='test_mapped_taskflow', start_date=days_ago(2)) as dag:
print(repr(value))
consumer.expand(value=make_list())
+ consumer.expand(value=[1,2,3])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
ashb commented on PR #22904:
URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095449301
Really? Those are failing for me on main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #22904: Call mapped_dependants only on the original task
Posted by GitBox <gi...@apache.org>.
ashb merged PR #22904:
URL: https://github.com/apache/airflow/pull/22904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org