You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/12/05 17:40:52 UTC
[PR] Always remove unecessary software from all github runners [datafusion]
alamb opened a new pull request, #19122:
URL: https://github.com/apache/datafusion/pull/19122
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes https://github.com/apache/datafusion/issues/19121
## Rationale for this change
- DataFusion requires a lot of diskspace (rust compilation, etc)
- Github often adds significant overhead as well with pre-installed software
- Some specific jobs remove unecessary pre-installed software, but it is not applied to all jobs (and so new jobs fail from time to time, such as https://github.com/apache/datafusion/issues/19121)
## What changes are included in this PR?
1. Let's apply the cleanup to all jobs
## Are these changes tested?
by CI
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#discussion_r2593468883
##########
.github/workflows/rust.yml:
##########
@@ -374,19 +362,6 @@ jobs:
with:
save-if: ${{ github.ref_name == 'main' }}
shared-key: "amd-ci-linux-test-example"
- - name: Remove unnecessary preinstalled software
- run: |
- echo "Disk space before cleanup:"
- df -h
- apt-get clean
- rm -rf /__t/CodeQL
Review Comment:
Deleting these individual tools is redundant now with
```shell
rm -rf /__t/* || true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618184350
I also manually triggered the extended test workflow:
- https://github.com/apache/datafusion/actions/runs/19973170431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618664829
> Thanks @alamb usually images tends to grow, hope it won't hit the limit soon, otherwise we would have to look for another images or potentially create own
Yeah, or cut down the size of the builds (@cj-zhukov has been helping for the examples)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618666215
Let's give it a try
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #19122:
URL: https://github.com/apache/datafusion/pull/19122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618666891
I am merging this one in to try and get the CI back to being stably green
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#discussion_r2593468883
##########
.github/workflows/rust.yml:
##########
@@ -374,19 +362,6 @@ jobs:
with:
save-if: ${{ github.ref_name == 'main' }}
shared-key: "amd-ci-linux-test-example"
- - name: Remove unnecessary preinstalled software
- run: |
- echo "Disk space before cleanup:"
- df -h
- apt-get clean
- rm -rf /__t/CodeQL
Review Comment:
covered with
```shell
rm -rf /__t/* || true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Always remove unecessary software from github runners for all jobs [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#discussion_r2593457994
##########
.github/workflows/rust.yml:
##########
@@ -272,18 +272,6 @@ jobs:
volumes:
- /usr/local:/host/usr/local
steps:
- - name: Remove unnecessary preinstalled software
Review Comment:
consolidated two separate cleanup lists into one
##########
.github/actions/setup-builder/action.yaml:
##########
@@ -46,3 +46,23 @@ runs:
# https://github.com/actions/checkout/issues/766
shell: bash
run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
+ - name: Remove unnecessary preinstalled software
Review Comment:
this actually probably slows down some jobs like fmt and clippy 🤔
Maybe we can have an explicit way to skip this on setup 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org