You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/12 22:04:46 UTC
[GitHub] [airflow] potiuk opened a new pull request, #25702: Add groups in CI output for parallell tasks
potiuk opened a new pull request, #25702:
URL: https://github.com/apache/airflow/pull/25702
Some of our tasks run in parallel in CI. Their output is now grouped
one group per parallel task and printed at the end only.
Progress is displayed while the tasks are executed.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25702: Add groups in CI output for parallell tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25702:
URL: https://github.com/apache/airflow/pull/25702#issuecomment-1216562707
Would be nice to get this one merged, seems to be green - this is yet another step to get rid of Bash breeze scripts as it unblocks #25678 and #23538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25702: Add groups in CI output for parallell tasks
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25702:
URL: https://github.com/apache/airflow/pull/25702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25702: Add groups in CI output for parallell tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25702:
URL: https://github.com/apache/airflow/pull/25702#issuecomment-1215876548
This one addresses the problem of having nice, usable output of optimized, parallel execution of our CI jobs using the new breeze:
* building CI images
* building PROD images
* pulling images
* building constraints
In a follow-up PR I will also switch remaining tests to use it: Docker Compose, Helm and finally Regular tests to use it (which should complete Breeze converstion to Python.
What we get is:
* no more problems with ci groups overlapping each-other. The output groups of Github are non-hierarchical (one level deep) and this change follow it nicely (no further grouping if logs are already in a group)
* nicer command foldable CI output
<img width="666" alt="Screenshot 2022-08-15 at 23 19 10" src="https://user-images.githubusercontent.com/595491/1
When folding expanded:
<img width="694" alt="Screenshot 2022-08-15 at 23 20 27" src="https://user-images.githubusercontent.com/595491/184720536-728996ae-6492-4dcd-8ee2-1d7bf42e4a7f.png">
* nice and "smart" progresss output while the parallel execution happens. It understands what kind of output is showing real progresss for the types of outputs we know (docker build/pull/pytest etc).
<img width="1354" alt="Screenshot 2022-08-15 at 23 31 33" src="https://user-images.githubusercontent.com/595491/184722126-1c0f24dc-ae0e-49ad-a720-9d4fea2c5d0a.png">
Including Pytest "%" progress
<img width="1424" alt="Screenshot 2022-08-15 at 23 32 45" src="https://user-images.githubusercontent.com/595491/184722375-868daaf7-f517-4877-8a25-3d5002812439.png">
* Folded "per-parallle-run" output that allows to see easily what happened in each run:
Folded:
<img width="563" alt="Screenshot 2022-08-15 at 23 24 40" src="https://user-images.githubusercontent.com/595491/184721524-2164f94d-462f-4ce1-95ee-343ebc2dd67c.png">
Unfolded PROD 3.7:
<img width="922" alt="Screenshot 2022-08-15 at 23 26 14" src="https://user-images.githubusercontent.com/595491/184721378-127da248-6576-40c0-b0c4-ac15991a74d2.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #25702: Add groups in CI output for parallell tasks
Posted by GitBox <gi...@apache.org>.
Bowrna commented on PR #25702:
URL: https://github.com/apache/airflow/pull/25702#issuecomment-1217521818
Thanks @potiuk I am merging this changes into branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25702: Add groups in CI output for parallell tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25702:
URL: https://github.com/apache/airflow/pull/25702#issuecomment-1216569860
cc: @Bowrna - once this one is merged, it should be easy to apply the shared "parallel" library approach to running tests in parallel (#23538) - it's really the matter of preparing List of outputs, and handling writing to a file if output is pased, instead of writing to stdout (and then adding method to show nicely the best one-line progrees from test outputs.
So once we merge it, if you would like to rebase your change on top of it and make use of it, this would be the best approach (I am going to do that with #25678)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org