You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/06/15 12:46:55 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue, #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
alamb opened a new issue, #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676
### Describe the bug
The CI on main is failing due to the runners running out of disk space
### To Reproduce
Example: https://github.com/apache/arrow-datafusion/actions/runs/5277391664/jobs/9545549038
```
"-Wl,-Bdynamic" "-llzma" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/__w/arrow-datafusion/arrow-datafusion/target/debug/deps/roundtrip_physical_plan-8dff8073998bbd8f" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
= note: /usr/bin/ld: final link failed: No space left on device
collect2: error: ld returned 1 exit status
error: could not compile `datafusion-substrait` (test "roundtrip_physical_plan") due to previous error
Error: Process completed with exit code 101.
```
### Expected behavior
Should pass
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1592979852
If someone else could look at this that would be great, otherwise I plan to do so in a few hours
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jonahgao commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1593235769
```shell
Run actions/checkout@v3
/usr/bin/docker exec fea1a1b1cd04085c[13](https://github.com/apache/arrow-datafusion/actions/runs/5277391664/jobs/9545549038#step:3:14)b1bc9e919041bcfd39ea946e15d5ad9c9b4373809eb9ce sh -c "cat /etc/*release | grep ^ID"
Syncing repository: apache/arrow-datafusion
Getting Git version info
Temporarily overriding HOME='/__w/_temp/a4e7f1c6-97f6-4dbc-9b18-a7b04a62be55' before making global git config changes
```
It could be caused by the incorrect home path.
`HOME='/__w/_temp/a4e7f1c6-97f6-4dbc-9b18-a7b04a62be55' `
The root directory usually has a small amount of space.
I saw a possibly related [issue](https://github.com/actions/runner/issues/2058).
But I don't know how to solve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1593454827
I will next sort out how to reduce the number and size of binaries
At least locally, for me, incremental compilation intermediate files seem to consume a large amount of space:
```
(arrow_dev) alamb@MacBook-Pro-8:~/Software/arrow-datafusion$ du -s -h target/debug/*
463M target/debug/build
1.6G target/debug/deps
780K target/debug/examples
4.2G target/debug/incremental
```
I will experiment with disabling that:
https://doc.rust-lang.org/cargo/reference/profiles.html?highlight=incremental#incremental
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1593444184
Based off of [this experiment](https://github.com/apache/arrow-datafusion/actions/runs/5280220590/jobs/9552021849?pr=6678):
Before any builds start the builder has 11G/35GB if its total disk, and has 33MB in the datafusion checkout:
```
11G /
33M /__w/arrow-datafusion/arrow-datafusion
```
After the build ends there is 35GB on the disk, all of it in the datafusion checkout:
```
35G /
24G /__w/arrow-datafusion/arrow-datafusion
```
Of that, almost all of it is in `target`:
```
4.0K /__w/arrow-datafusion/arrow-datafusion/CHANGELOG.md
4.0K /__w/arrow-datafusion/arrow-datafusion/CODE_OF_CONDUCT.md
4.0K /__w/arrow-datafusion/arrow-datafusion/CONTRIBUTING.md
108K /__w/arrow-datafusion/arrow-datafusion/Cargo.lock
4.0K /__w/arrow-datafusion/arrow-datafusion/Cargo.toml
12K /__w/arrow-datafusion/arrow-datafusion/LICENSE.txt
4.0K /__w/arrow-datafusion/arrow-datafusion/NOTICE.txt
4.0K /__w/arrow-datafusion/arrow-datafusion/README.md
224K /__w/arrow-datafusion/arrow-datafusion/benchmarks
24K /__w/arrow-datafusion/arrow-datafusion/ci
44K /__w/arrow-datafusion/arrow-datafusion/conbench
13M /__w/arrow-datafusion/arrow-datafusion/datafusion
196K /__w/arrow-datafusion/arrow-datafusion/datafusion-cli
160K /__w/arrow-datafusion/arrow-datafusion/datafusion-examples
792K /__w/arrow-datafusion/arrow-datafusion/dev
444K /__w/arrow-datafusion/arrow-datafusion/docs
4.0K /__w/arrow-datafusion/arrow-datafusion/header
2.7M /__w/arrow-datafusion/arrow-datafusion/parquet-testing
4.0K /__w/arrow-datafusion/arrow-datafusion/pre-commit.sh
8.0K /__w/arrow-datafusion/arrow-datafusion/python
4.0K /__w/arrow-datafusion/arrow-datafusion/rustfmt.toml
24G /__w/arrow-datafusion/arrow-datafusion/target. <-- here is the problem
28K /__w/arrow-datafusion/arrow-datafusion/test-utils
8.8M /__w/arrow-datafusion/arrow-datafusion/testing
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1593623954
LOL well I have some fixes that I think will give us some breathing room for a while:
- [ ] https://github.com/apache/arrow-datafusion/pull/6688 (Save 2GB)
- [ ] https://github.com/apache/arrow-datafusion/pull/6682 (save 125MB)
- [ ] https://github.com/apache/arrow-datafusion/pull/6683 (save 250MB)
- [ ] https://github.com/apache/arrow-datafusion/pull/6685 (save 125MB)
- [ ] https://github.com/apache/arrow-datafusion/pull/6684 (save 1GB)
- [ ] https://github.com/apache/arrow-datafusion/pull/6686 (save 250MB)
(a total of 3.75GB :bowtie:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1593485461
`incremental` seems to take 2GB of space on the builder -- will try disabling it and see what happens
https://github.com/apache/arrow-datafusion/actions/runs/5281745040/jobs/9555736433?pr=6678
```
Run du -s -h $GITHUB_WORKSPACE/target/debug/* || true
429M /__w/arrow-datafusion/arrow-datafusion/target/debug/build
21G /__w/arrow-datafusion/arrow-datafusion/target/debug/deps
4.0K /__w/arrow-datafusion/arrow-datafusion/target/debug/examples
2.3G /__w/arrow-datafusion/arrow-datafusion/target/debug/incremental
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6676:
URL: https://github.com/apache/arrow-datafusion/issues/6676#issuecomment-1594883050
After applying all our the updates from yesterday, I measured disk usage again: https://github.com/apache/arrow-datafusion/actions/runs/5291555219/jobs/9577253624?pr=6678
Before the work the disk had 35GB at the end of the test run
```
35G /
```
After all the changes it has 8GB to spare
```
27G /
```
🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6676: CI is failing due to "/usr/bin/ld: final link failed: No space left on device"
URL: https://github.com/apache/arrow-datafusion/issues/6676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org