You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "SteveLauC (via GitHub)" <gi...@apache.org> on 2024/02/29 03:37:40 UTC
[I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
SteveLauC opened a new issue, #9398:
URL: https://github.com/apache/arrow-datafusion/issues/9398
### Is your feature request related to a problem or challenge?
As a contributor, I would like to test more to reduce CI errors, I tried running `cargo test` at the project root, and then all my RAM was eaten by tests, and the kernel got killed, forcing me to restart my PC.
I have 32 GB of memory, looks like I need at least 64 GB or 128 GB :D
### Describe the solution you'd like
Document the spec (minimal one) needed for running tests of DataFusion in the contributors' guide.
And, we should document which tests are resource-heavy so that contributors can ignore them locally.
### Describe alternatives you've considered
No
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #9398:
URL: https://github.com/apache/arrow-datafusion/issues/9398#issuecomment-1971022831
Glad that worked for you! I opened #9402 to document this in the contributors guide.
There does not seem to be a way to configure Cargo to default to `--test-threads 1` other than setting environment variables, so documentation may be our best bet for now... see https://github.com/rust-lang/cargo/issues/8430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
Posted by "SteveLauC (via GitHub)" <gi...@apache.org>.
SteveLauC commented on issue #9398:
URL: https://github.com/apache/arrow-datafusion/issues/9398#issuecomment-1970971223
> cargo test -- --test-threads=1
Thanks for showing me this! This indeed makes the memory usage controllable:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #9398:
URL: https://github.com/apache/arrow-datafusion/issues/9398#issuecomment-1970506850
I think the doctests are the main culprit, see #5347
Agree on documenting this as part of contributor/developer guide 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9398:
URL: https://github.com/apache/arrow-datafusion/issues/9398#issuecomment-1971336566
I had my computer upgraded to 64GB of ram - the workaround mentioned above I used as well but the time to run the tests was using that was horrendous. Using linux under WSL2 in windows here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
Posted by "devinjdangelo (via GitHub)" <gi...@apache.org>.
devinjdangelo commented on issue #9398:
URL: https://github.com/apache/arrow-datafusion/issues/9398#issuecomment-1970956507
Thank you for bringing this up @SteveLauC. I had the same experience when I first starting working on DataFusion. You can work around memory limitations by running `cargo test -- --test-threads=1` which will only run a single test at a time. It will be slower but consume substantially less memory. I think it would be a good idea to document this workaround since many new contributors won't have enough RAM to run all tests at max parallelism on their system.
The precise memory requirements to run `cargo test` will vary over time and depend on your exact development set up. Running in `wsl` on Windows for example is a bit more intensive as you need to reserve some memory for Windows. I personally upgraded from 32->64GB and that was plenty running natively on linux.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Document the minimal resources needed to run tests for DataFusion [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #9398: Document the minimal resources needed to run tests for DataFusion
URL: https://github.com/apache/arrow-datafusion/issues/9398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org