You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/28 19:59:14 UTC

[GitHub] [airflow] dstandish opened a new pull request, #27971: Experiment: Run static checks and postgres tests before all else

dstandish opened a new pull request, #27971:
URL: https://github.com/apache/airflow/pull/27971

   This should allow for faster failure / iteration time on non-hosted runners.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #27971:
URL: https://github.com/apache/airflow/pull/27971#issuecomment-1329823364

   yeah i agree.  i was actually just about to close the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27971:
URL: https://github.com/apache/airflow/pull/27971#discussion_r1034095249


##########
.github/workflows/ci.yml:
##########
@@ -778,8 +780,8 @@ jobs:
     needs: [build-info, wait-for-ci-images]
     strategy:
       matrix:
-        python-version: "${{fromJson(needs.build-info.outputs.python-versions)}}"
-        postgres-version: "${{fromJson(needs.build-info.outputs.postgres-versions)}}"
+        python-version: ["3.7"]

Review Comment:
   This is a bit dangerous and not needed. For most cases (as of last week) the exclude matrices are only selecting "representative" combos for tests - so you do not save by hard-conding those I think.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27971:
URL: https://github.com/apache/airflow/pull/27971#discussion_r1034114562


##########
.github/workflows/ci.yml:
##########
@@ -778,8 +780,8 @@ jobs:
     needs: [build-info, wait-for-ci-images]
     strategy:
       matrix:
-        python-version: "${{fromJson(needs.build-info.outputs.python-versions)}}"
-        postgres-version: "${{fromJson(needs.build-info.outputs.postgres-versions)}}"
+        python-version: ["3.7"]

Review Comment:
   yeah both this and putting `test-postgres` everywhere was *just* for testing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on a diff in pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
dstandish commented on code in PR #27971:
URL: https://github.com/apache/airflow/pull/27971#discussion_r1034103825


##########
.github/workflows/ci.yml:
##########
@@ -366,7 +368,7 @@ jobs:
       ${{needs.build-info.outputs.build-job-description}} PROD images
       ${{needs.build-info.outputs.all-python-versions-list-as-string}}
     runs-on: "${{needs.build-info.outputs.runs-on}}"
-    needs: [build-info, build-ci-images]
+    needs: [build-info, build-ci-images, tests-postgres]

Review Comment:
   i agree, if we want to do something like that... this would be the way to go.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #27971:
URL: https://github.com/apache/airflow/pull/27971#issuecomment-1329825428

   it would help if there were too many PRs competing for too few runners -- then we quickly get a quick read on the tests most likely to fail.  but that doesn't really seem to be the case here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27971:
URL: https://github.com/apache/airflow/pull/27971#discussion_r1034097967


##########
.github/workflows/ci.yml:
##########
@@ -366,7 +368,7 @@ jobs:
       ${{needs.build-info.outputs.build-job-description}} PROD images
       ${{needs.build-info.outputs.all-python-versions-list-as-string}}
     runs-on: "${{needs.build-info.outputs.runs-on}}"
-    needs: [build-info, build-ci-images]
+    needs: [build-info, build-ci-images, tests-postgres]

Review Comment:
   I suggest to introduce an empty "smoke-tests" job that will be dependent o tests-postgres and use dependency like.
   
   ```
   [build-info, smoke-tests]
   ```
   
   This will be much more "generic". Unlike build-info (which is used in "needs.build-info", we do not need `build-ci-images` if "smoke-tests" (or "tests-postgres" depend on it already. Unfortunately github actions dependencies are not transitive and if we need "build-info" outputs, in `needs` clause we need to add it here.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish closed pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
dstandish closed pull request #27971: Experiment: Run static checks and postgres tests before all else
URL: https://github.com/apache/airflow/pull/27971


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27971:
URL: https://github.com/apache/airflow/pull/27971#issuecomment-1329820819

   I am not sure if this is helpful,
   
   For non-hosted runners it should be no difference. We already have 900 runners by the ASF and we should have no more queueing here (at least we stopped seeing those) - so this is pretty ok to run all test types in parallel. And it will introduce significant "sequential" delays in case of non-postgres failures I am afraid. Rather than running tests in parallel, it will run them sequentially so "feedback time" will be biggeer.
   
   When optimising CI builds there are two aspects:
   
   * feedback time
   * cost (build hours usage)
   
   Feedback time is very important - the  time between push and "result" should be as short as possible. Putting sequential dependency between tests increases feedback time, but saves cost (assuming job queue is infinite - and with 900 jobs in the ASF as of 3 weeks, for now - until ASF usage grows significantly it almost is - we used to have 300) . Cost does not matter - it is essentially free.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish closed pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
dstandish closed pull request #27971: Experiment: Run static checks and postgres tests before all else
URL: https://github.com/apache/airflow/pull/27971


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #27971: Experiment: Run static checks and postgres tests before all else

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27971:
URL: https://github.com/apache/airflow/pull/27971#discussion_r1034097967


##########
.github/workflows/ci.yml:
##########
@@ -366,7 +368,7 @@ jobs:
       ${{needs.build-info.outputs.build-job-description}} PROD images
       ${{needs.build-info.outputs.all-python-versions-list-as-string}}
     runs-on: "${{needs.build-info.outputs.runs-on}}"
-    needs: [build-info, build-ci-images]
+    needs: [build-info, build-ci-images, tests-postgres]

Review Comment:
   I suggest to introduce an empty "smoke-tests" job that will be dependent o tests-postgres and use dependency like.
   
   ```
   [build-info, smoke-tests]
   ```
   
   This will be much more "generic". Unlike build-info (which is used in "needs.build-info", we do not need `build-ci-images" if "smoke-tests" (or "tests-postgres" depend on it already. Unfortunately github actions dependencies are not transitive and if we need "build-info" outputs, in `needs` clause we need to add it here.
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org