You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/12/05 17:40:52 UTC

[PR] Always remove unecessary software from all github runners [datafusion]

alamb opened a new pull request, #19122:
URL: https://github.com/apache/datafusion/pull/19122

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes https://github.com/apache/datafusion/issues/19121
   
   ## Rationale for this change
   
   - DataFusion requires a lot of diskspace (rust compilation, etc)
   - Github often adds significant overhead as well with pre-installed software
   - Some specific jobs remove unecessary pre-installed software, but it is not applied to all jobs (and so new jobs fail from time to time, such as  https://github.com/apache/datafusion/issues/19121)
   
   ## What changes are included in this PR?
   1. Let's apply the cleanup to all jobs
   
   ## Are these changes tested?
   by CI
   
   ## Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#discussion_r2593468883


##########
.github/workflows/rust.yml:
##########
@@ -374,19 +362,6 @@ jobs:
         with:
           save-if: ${{ github.ref_name == 'main' }}
           shared-key: "amd-ci-linux-test-example"
-      - name: Remove unnecessary preinstalled software
-        run: |
-          echo "Disk space before cleanup:"
-          df -h
-          apt-get clean
-          rm -rf /__t/CodeQL

Review Comment:
   Deleting these individual tools is redundant now with
   ```shell
             rm -rf /__t/* || true
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618184350

   I also manually triggered the extended test workflow: 
   - https://github.com/apache/datafusion/actions/runs/19973170431


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618664829

   > Thanks @alamb usually images tends to grow, hope it won't hit the limit soon, otherwise we would have to look for another images or potentially create own
   
   Yeah, or cut down the size of the builds (@cj-zhukov  has been helping for the examples)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618666215

   Let's give it a try


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #19122:
URL: https://github.com/apache/datafusion/pull/19122


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#issuecomment-3618666891

   I am merging this one in to try and get the CI back to being stably green


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#discussion_r2593468883


##########
.github/workflows/rust.yml:
##########
@@ -374,19 +362,6 @@ jobs:
         with:
           save-if: ${{ github.ref_name == 'main' }}
           shared-key: "amd-ci-linux-test-example"
-      - name: Remove unnecessary preinstalled software
-        run: |
-          echo "Disk space before cleanup:"
-          df -h
-          apt-get clean
-          rm -rf /__t/CodeQL

Review Comment:
   covered with
   ```shell
             rm -rf /__t/* || true
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Always remove unecessary software from github runners for all jobs [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19122:
URL: https://github.com/apache/datafusion/pull/19122#discussion_r2593457994


##########
.github/workflows/rust.yml:
##########
@@ -272,18 +272,6 @@ jobs:
       volumes:
         - /usr/local:/host/usr/local
     steps:
-      - name: Remove unnecessary preinstalled software

Review Comment:
   consolidated two separate cleanup lists into one



##########
.github/actions/setup-builder/action.yaml:
##########
@@ -46,3 +46,23 @@ runs:
       # https://github.com/actions/checkout/issues/766
       shell: bash
       run: git config --global --add safe.directory "$GITHUB_WORKSPACE"
+    - name: Remove unnecessary preinstalled software

Review Comment:
   this actually probably slows down some jobs like fmt and clippy 🤔 
   
   Maybe we can have an explicit way to skip this on setup 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org