You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/08/14 16:18:22 UTC
[I] `cargo test --test fuzz` fails with "Too many open files" [datafusion]
alamb opened a new issue, #17194:
URL: https://github.com/apache/datafusion/issues/17194
### Describe the bug
- While verifying 49.0.1: https://github.com/apache/datafusion/issues/17036
@milenkovicm reported ([email](https://lists.apache.org/thread/g94ztqk5hlgxd0lnmhofsm5lowchnn7c))
> I’m getting
```
---- fuzz_cases::sort_fuzz::test_sort_10k_mem stdout ----
thread 'fuzz_cases::sort_fuzz::test_sort_10k_mem' panicked at datafusion/core/tests/fuzz_cases/sort_fuzz.rs:269:63:
called `Result::unwrap()` on an `Err` value: Execution("Failed to create partition file at \"/var/folders/h_/cdg2zx090212402xhrwhv62w0000gn/T/.tmpdhEV7B/.tmpIhvMHc\": Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
fuzz_cases::sort_fuzz::test_sort_10k_mem
test result: FAILED. 75 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 73.00s
```
### To Reproduce
In a clean terminal without any adjusted `ulimit` run
```shell
cargo test --test fuzz
```
And you'll see
```
---- fuzz_cases::sort_fuzz::test_sort_10k_mem stdout ----
thread 'fuzz_cases::sort_fuzz::test_sort_10k_mem' panicked at datafusion/core/tests/fuzz_cases/sort_fuzz.rs:269:63:
called `Result::unwrap()` on an `Err` value: Execution("Failed to create partition file at \"/var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/.tmpj128U3/.tmp86cD6k\": Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_aggregate_with_high_cardinality_with_limited_memory_and_different_sizes_of_record_batch_and_changing_memory_reservation stdout ----
Error: Execution("Failed to create partition file at \"/var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/.tmpLsX6yT/.tmpHO0Q6l\": Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
---- fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_aggregate_with_high_cardinality_with_limited_memory_and_large_record_batch stdout ----
Error: Execution("Failed to create partition file at \"/var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/.tmpRtUy2o/.tmp8OqkYC\": Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
---- fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_sort_with_limited_memory_and_large_record_batch stdout ----
Error: IoError(Custom { kind: Uncategorized, error: PathError { path: "/var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/.tmp7aaX1I/.tmpX6Jzr0", err: Os { code: 24, kind: Uncategorized, message: "Too many open files" } } })
---- fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_aggregate_with_high_cardinality_with_limited_memory_and_different_sizes_of_record_batch_and_take_all_memory stdout ----
Error: Execution("Failed to create partition file at \"/var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/.tmp0MNZTo/.tmp31Qukn\": Os { code: 24, kind: Uncategorized, message: \"Too many open files\" }")
failures:
fuzz_cases::sort_fuzz::test_sort_10k_mem
fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_aggregate_with_high_cardinality_with_limited_memory_and_different_sizes_of_record_batch_and_changing_memory_reservation
fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_aggregate_with_high_cardinality_with_limited_memory_and_different_sizes_of_record_batch_and_take_all_memory
fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_aggregate_with_high_cardinality_with_limited_memory_and_large_record_batch
fuzz_cases::spilling_fuzz_in_memory_constrained_env::test_sort_with_limited_memory_and_large_record_batch
test result: FAILED. 81 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out; finished in 81.65s
```
### Expected behavior
I would really like the tests to run successfully with the default `ulimits` on Mac. It seems the default is 256 on my machine:
> open files (-n) 256
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8176
cpu time (seconds, -t) unlimited
max user processes (-u) 10666
virtual memory (kbytes, -v) unlimited
```
### Additional context
The workaround / way to fix this is to raise the limit on the number of files via `ulimit`
For example, setting the open file limit to 4096 beforehand will solve the problem:
```shell
ulimit -n 4096
cargo test --test fuzz
```
Passes
```
test result: ok. 86 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 73.86s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `cargo test --test fuzz` fails with "Too many open files" [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed issue #17194: `cargo test --test fuzz` fails with "Too many open files"
URL: https://github.com/apache/datafusion/issues/17194
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org