You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/28 08:20:35 UTC

[GitHub] [airflow] bhirsz opened a new pull request, #26742: Fix GCS sensor system tests failing with DebugExecutor

bhirsz opened a new pull request, #26742:
URL: https://github.com/apache/airflow/pull/26742

   Thanks to ping from @ syedahsn  I have noticed that one of our GCS sensors fails when run in system tests with DebugExecutor (we use other executors in our CI that's why I didn't notice it). 
   
   This PR provides a workaround for this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] bhirsz commented on pull request #26742: Fix GCS sensor system tests failing with DebugExecutor

Posted by GitBox <gi...@apache.org>.
bhirsz commented on PR #26742:
URL: https://github.com/apache/airflow/pull/26742#issuecomment-1262758926

   > I know this was already merged by I have some questions:
   > 
   > * Is there any risk to forcing the sensor to be always in Poke mode? (If it was determined by the original author to not be runnable in that mode)
   > * I'm curious what executor you folks use for your system tests, can you explain anything more about that :)
   
   1) I think this page explains it the best: https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html in our particular case we force poke mode only for one sensor and typically it's up to user to chose the mode (I think so). However I'm not entirely sure if this sensor couldn't be implemented in a way that would allow reschedule mode. But it would require some extra analysis
   
   2) well, you can take any example dag and run it any deployed Airflow like any other dag (granted you have copied test utils as well). In our case we deploy our Airflow image to test env and copy test Dags there. We also have monitoring dag which check whether other dags finished and when they do it save this information. Our test framework then generates dynamic test based on list of dags and each test polls for the information whether given dag completes (and with what status and logs). It can use any executor, for example KubernetesExecutor. 
   Locally I'm testing either using breeze, or by deploying dag to airflow, or by using this CI based on cloud build: https://github.com/bhirsz/airflow-system-tests
   It will spawn 1 cloud build for each example dag and run it using pytest. 
   
   It's not done but I hope to finish it in the close future - I'm trying to return to system testing since I was involved in other task recently. 
   
   What do you use to run system tests?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #26742: Fix GCS sensor system tests failing with DebugExecutor

Posted by GitBox <gi...@apache.org>.
potiuk merged PR #26742:
URL: https://github.com/apache/airflow/pull/26742


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] o-nikolas commented on pull request #26742: Fix GCS sensor system tests failing with DebugExecutor

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on PR #26742:
URL: https://github.com/apache/airflow/pull/26742#issuecomment-1262667896

   I know this was already merged by I have some questions:
   
   - Is there any risk to forcing the sensor to be always in Poke mode? (If it was determined by the original author to not be runnable in that mode)
   - I'm curious what executor you folks use for your system tests, can you explain anything more about that :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org