You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/09 17:48:23 UTC

[GitHub] [airflow] potiuk opened a new pull request #18853: Fix occassional external task sensor test failures

potiuk opened a new pull request #18853:
URL: https://github.com/apache/airflow/pull/18853


   Occassionally the sensor tests fail with assertion where
   state seems to be None. This might be caused by
   
   ```
         def assert_ti_state_equal(task_instance, state):
             """
             Assert state of task_instances equals the given state.
             """
             task_instance.refresh_from_db()
     >       assert task_instance.state == state
     E       AssertionError: assert None == <TaskInstanceState.SUCCESS: 'success'>
     E        +  where None = <TaskI$anstance: dag_1.task_b_1 manual__2015-01-01T00:00:00+00:00 [None]>.state
   ```
   
   Turned out it was because the task instance fields from
   dagrun.taskinstance relationship could be returned in different
   order so some of the dependencies were not met for some of the
   tasks when later task was returned before earlier one.
   
   Deterministic sorting according to task_id solved the problem.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939362338


   > Deterministic sorting according to task_id solved the problem.
   
   Why? The dependency-based ordering (topological) does not guarantee to match alphabetical ordering. I think we need to actually parse the dependencies to determine ordering...?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939364036


   > > Why? The dependency-based ordering (topological) does not guarantee to match alphabetical ordering. I think we need to > Because this is how test dags are constructed. If you process them in the deterministic order, it works. If the order is random, it sometimes does not. That's as simple as that. there is no "dependency check" here. Unlike the previous attemtpt which sometimes worked, sometimes did not - this will work for sure.
   
   The problem (here and few other tests) is that there was an implicit assumption that all tasks will have state set. But this is not how it works when you process the tasks in test in random order. If you first process the tasks that are depending on other (non-processed) tasks, they will keeep their state to None - which causes test failure. Fortunately the structure of the test tasks is that if you process them in the alphabetical order, the test will always succeed - because task1 is before task2 (and in our tests task2 depends on task1). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939368449


   @uranusjr  I've added a comment that explains why we've used the regular sort rather than topological sort (with our test DAG structure those two are equivalent)- same here "simple" is better than "right" especially that we are in full control of the "input" (DAG structure) and we do not have to implement a general solution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #18853:
URL: https://github.com/apache/airflow/pull/18853


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil merged pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #18853:
URL: https://github.com/apache/airflow/pull/18853


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939364036


   > > Why? The dependency-based ordering (topological) does not guarantee to match alphabetical ordering. I think we need to > Because this is how test dags are constructed. If you process them in the deterministic order, it works. If the order is random, it sometimes does not. That's as simple as that. there is no "dependency check" here. Unlike the previous attemtpt which sometimes worked, sometimes did not - this will work for sure.
   
   The problem (here and few other tests) is that there was an implicit assumption that all tasks will have state set. But this is not how it works when you process the tasks in test in random order. If you first process the tasks that are depending on other (non-processed) tasks, they will keeep their state to None - which causes test failure. Fortunately the structure of the test tasks is that if you process them in the alphabetical order, the test will always succeed - because task1 is before task2 (and in our tests tas2 depends on task1). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939364036


   > > Why? The dependency-based ordering (topological) does not guarantee to match alphabetical ordering. I think we need to > Because this is how test dags are constructed. If you process them in the deterministic order, it works. If the order is random, it sometimes does not. That's as simple as that. there is no "dependency check" here. Unlike the previous attemtpt which sometimes worked, sometimes did not - this will work for sure.
   
   The problem (here and few other tests) is that there was an implicit assumption that all tasks will have state set. But this is not how it works when you process the tasks in test in random error. If you first process the tasks that are depending on other (non-processed) tasks, they will keeep their state to None - which causes test failure. Fortunately the structure of the test tasks is that if you process them in the alphabetical order, the test will always succeed - because task1 is before task2 (and in our tests tas2 depends on task1). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #18853:
URL: https://github.com/apache/airflow/pull/18853


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939364700


   BTW. I woudl be happy to process tasks in all the tests in topological order and maybe we can add a tool that all tests will do to do so, but IMHO it's quite complex piece of code, and I am on a quest to fix the flaky tests with least amount of effort.
   I think if we can stabilize the tests with one line change, basing it on the fact that our tests structure is "stable" for alphabethical orded (because this is how our test DAGs are constructed)  it's a HUGE win for the community :).
   
   Unless of course there is other better way to do it without heavy investments (and I am all ears for that).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-939363437


    > Why? The dependency-based ordering (topological) does not guarantee to match alphabetical ordering. I think we need to actually parse the dependencies to determine ordering...?
   
   Because this is how test dags are constructed. If you process them in the deterministic order, it works. If the order is random, it sometimes does not. That's as simple as that. there is no "dependency check" here. Unlike the previous attemtpt which sometimes worked, sometimes did not - this will work for sure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #18853:
URL: https://github.com/apache/airflow/pull/18853#issuecomment-940943132


   Same here - I saw this test failing in at least 3-4 PRs recently. Merging it would be helpful to avoid false negatives.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil merged pull request #18853: Fix occassional external task sensor test failures

Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #18853:
URL: https://github.com/apache/airflow/pull/18853


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org