You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@netbeans.apache.org by GitBox <gi...@apache.org> on 2022/03/31 00:54:50 UTC
[GitHub] [netbeans] mbien opened a new pull request #3901: ci pipeline improvements
mbien opened a new pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901
draft PR to test some optimizations.
idea1: use Zstandard compression for pipeline artifacts.
- faster compression (multithreaded) and extraction compared to gz
- smaller artifact footprint (with `-9` setting, seems to be the sweet spot in my experience from other deployments)
- all github nodes support it [out of the box](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#preinstalled-software)
- cumulative performance improvement due to the nature of the pipeline
Local test using 2 threads to simulate a gh node. Tested in a ramdisk. The actual code uses `-T0` to be forward compatible.
```bash
# compression
time tar -czf /tmp/build.tar.gz --exclude ".git" .
real 1m12,641s
user 1m18,657s
sys 0m3,167s
# uses two threads to simmulate CI node
time tar -I 'zstd -9 -T2' -cf /tmp/build.tar.zst --exclude ".git" .
real 0m22,993s
user 0m50,221s
sys 0m3,018s
# archive sizes
ls -l /tmp/build*
-rw-r--r-- 1 mbien mbien 1575854860 31. Mär 01:21 /tmp/build.tar.gz
-rw-r--r-- 1 mbien mbien 1431884800 31. Mär 01:25 /tmp/build.tar.zst
# extraction
time tar -xzf /tmp/build.tar.gz
real 0m14,681s
user 0m15,051s
sys 0m3,438s
time tar --zstd -xf /tmp/build.tar.zst
real 0m3,747s
user 0m2,399s
sys 0m3,006s
```
outside of gh actions I would have combined compression + upload in one step (same for download + extraction). But this isn't possible while using the default upload action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job, look at windows :)
```
Run tar --zstd -xf build.tar.zst
tar: Option --zstd is not supported
Usage:
List: tar -tf <archive-filename>
Extract: tar -xf <archive-filename>
Create: tar -cf <archive-filename> [filenames...]
Help: tar --help
Error: Process completed with exit code 1.
```
results:
- compression down from 1m 42s to 53s
- extraction down from 34s to 16s.
- mac has to use `unzsd | tar` as workaround
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job, look at windows.
```
Run tar --zstd -xf build.tar.zst
tar: Option --zstd is not supported
Usage:
List: tar -tf <archive-filename>
Extract: tar -xf <archive-filename>
Create: tar -cf <archive-filename> [filenames...]
Help: tar --help
Error: Process completed with exit code 1.
```
results:
- compression down from 1m 42s to 53s
- extraction down from 34s to 16s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien commented on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien commented on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1086580342
> Thank you - the final part (not deleting artifacts on failure) is IMHO the most important I had to restart the full pipeline to fix two jobs, which kind of defeats the purpose.
Agreed. Thanks for the review. Lets get this merged so that job restarts can be used as advertised in the original PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job, look at windows.
results:
- compression down from 1m 42s to 1m 2s
- extraction down from 34s to 16s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien commented on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien commented on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job, look at windows :)
```
Run tar --zstd -xf build.tar.zst
tar: Option --zstd is not supported
Usage:
List: tar -tf <archive-filename>
Extract: tar -xf <archive-filename>
Create: tar -cf <archive-filename> [filenames...]
Help: tar --help
Error: Process completed with exit code 1.
```
results:
- compression down from 1m 42s to 53s
- extraction down from 34s to 16s.
- mac needs a workaround
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job, look at windows :)
```
Run tar --zstd -xf build.tar.zst
tar: Option --zstd is not supported
Usage:
List: tar -tf <archive-filename>
Extract: tar -xf <archive-filename>
Create: tar -cf <archive-filename> [filenames...]
Help: tar --help
Error: Process completed with exit code 1.
```
results:
- compression down from 1m 42s to 53s
- extraction down from 34s to 16s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien merged pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien merged pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job.
results:
- compression down from 1m 42s to 1m 2s
- extraction down from 34s to 16s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
[GitHub] [netbeans] mbien edited a comment on pull request #3901: ci pipeline improvements
Posted by GitBox <gi...@apache.org>.
mbien edited a comment on pull request #3901:
URL: https://github.com/apache/netbeans/pull/3901#issuecomment-1083922453
MacOS, please do your job, look at windows.
```
Run tar --zstd -xf build.tar.zst
tar: Option --zstd is not supported
Usage:
List: tar -tf <archive-filename>
Extract: tar -xf <archive-filename>
Create: tar -cf <archive-filename> [filenames...]
Help: tar --help
Error: Process completed with exit code 1.
```
results:
- compression down from 1m 42s to 1m 2s
- extraction down from 34s to 16s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@netbeans.apache.org
For additional commands, e-mail: notifications-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists