You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/10 08:03:29 UTC
[GitHub] [arrow-rs] tustvold opened a new issue, #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
tustvold opened a new issue, #1833:
URL: https://github.com/apache/arrow-rs/issues/1833
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Since updating MIRI in #1828 it is periodically OOMing - https://github.com/apache/arrow-rs/actions/workflows/miri.yaml
https://github.com/apache/arrow-rs/actions/runs/2473012537
It has also been a long-time annoyance of mine that
**Describe the solution you'd like**
I'm not entirely sure what the best course of action here is, rolling back to a 6 month old MIRI is not ideal and would require backing out changes in #1822, but then neither is having it randomly fail.
It has been a long-time annoyance of mine that the the CI currently takes ~40 minutes to chug through, despite significant caching. This is largely because the runners are rather piddly - https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources. This also precludes automatically running any meaningful benchmarks (#1274). Perhaps we should invest some time into a more powerful CI system such as [buildkite](https://buildkite.com/apache-arrow) which is used by other arrow projects...
**Describe alternatives you've considered**
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #1833:
URL: https://github.com/apache/arrow-rs/issues/1833#issuecomment-1425910833
The improvements to split up the CI have largely addressed this, so closing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1833:
URL: https://github.com/apache/arrow-rs/issues/1833#issuecomment-1152216425
other parts of the arrow project (e.g. C++) was talking about this -- there may be more beefy infrastructure we could take advantage of there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #1833:
URL: https://github.com/apache/arrow-rs/issues/1833#issuecomment-1177969640
Another dimension this may provide improvement is disk space, with the 14GB proving insufficient for some use-cases (#2004)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jhorstmann commented on issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by GitBox <gi...@apache.org>.
jhorstmann commented on issue #1833:
URL: https://github.com/apache/arrow-rs/issues/1833#issuecomment-1153734180
I recently noticed some of the fuzzing tests for filters running for a long time. Watching `htop` also showed memory usage increasing while running `fuzz_filter`. Memory usage was already at ~8Gb before that test though.
Maybe some of those tests could be excluded from running in the miri cfg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by GitBox <gi...@apache.org>.
viirya commented on issue #1833:
URL: https://github.com/apache/arrow-rs/issues/1833#issuecomment-1177991040
Recently I saw many times of running out of disk space in CI. Although it seems better now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1833:
URL: https://github.com/apache/arrow-rs/issues/1833#issuecomment-1193193669
I got inspired this afternoon while waiting for some other PRs to finish up CI and started a few improvements using the github runners via https://github.com/apache/arrow-rs/issues/2149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #1833: Larger CI Runners to Prevent MIRI OOMing and Improve CI Times
URL: https://github.com/apache/arrow-rs/issues/1833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org