You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "cetra3 (via GitHub)" <gi...@apache.org> on 2026/02/05 03:09:18 UTC
[PR] Fix Arrow Spill Underrun [datafusion]
cetra3 opened a new pull request, #20159:
URL: https://github.com/apache/datafusion/pull/20159
## Which issue does this PR close?
Closes https://github.com/apache/datafusion/issues/19425
## Rationale for this change
This adjusts the way that the spill channel works. Currently we have a spill writer & reader pairing which uses a mutex to coordindate when a file is ready to be read.
What happens is, that because we were using a `spawn_buffered` call, the read task would race ahead trying to read a file which is yet to be written out completely.
Alongside this, we need to flush each write to the file, as there is a chance that another thread may see stale data.
## What changes are included in this PR?
Adds a flush on write, and converts the read task to not buffer reads.
## Are these changes tested?
I haven't written a test, but I have been running the example in the attached issue. While it now fails with allocation errors, the original error goes away.
## Are there any user-facing changes?
Nope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3856866324
Iβd guess a file IO per batch is okay. A file per batch probably not thatβs what all of this complexity tries to avoid). But it would be interesting to (1) confirm this PR doesnβt make other spilling queries slower and (2) investigate why the spilling version is so slow (can be done outside of this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3993052023
Makes sense, thanks for the explainer @hareshkh. I think in principle we'd love to provide more minor releases for reasons exactly like this! I expect that as DataFusion grows and gets used by more and more projects, and this problem of multi-tiered dependencies will only get worse. Since you've helped organize releases a bit already, would you / Palantir be willing to drive and organize this release? It would be much appreciated ππ»
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3858307087
Here are the results from the clickbench benchmark. With 1GB memory limit, however we reach the resource limits but the bug doesn't materialize with the change:
[fix_spill_read_underrun.log](https://github.com/user-attachments/files/25119991/fix_spill_read_underrun.log)
[main.log](https://github.com/user-attachments/files/25119993/main.log)
[clickbench_partitioned.json -fix](https://github.com/user-attachments/files/25119997/clickbench_partitioned.json)
[clickbench_partitioned.json - main](https://github.com/user-attachments/files/25120001/clickbench_partitioned.json)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3885722084
run benchmark tpch_mem
DATAFUSION_RUNTIME_MEMORY_LIMIT=1G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "hareshkh (via GitHub)" <gi...@apache.org>.
hareshkh commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3992502694
> > @cetra3 @adriangb @alamb : Is there any interest in a 52.3.0 release? This bug is causing some problems for us. I am happy to create an issue/backports etc.
>
> The 53.0.0 release is going to be made soon - could you upgrade to that?
@adriangb: Yes, 53.0.0 would work too, but we also depend on datafusion-comet, and the 53 release will take longer to get there before we can take it in. Since this causes correctness issues at runtime, I am keen if there is a chance this can be released as 52.3.0 as well, while updating Comet to use 52.3.0 should be a relatively lower lift.
This is another PR around the same fix - https://github.com/apache/datafusion/pull/20672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3885943088
run benchmark clickbench
DATAFUSION_RUNTIME_MEMORY_LIMIT=500M
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3857068999
@alamb Wdyt of adding a benchmark that forces spilling? Or an option to the current script to run with a memory limit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3859913330
> Here are the results from the clickbench benchmark. With 1GB memory limit, however we reach the resource limits but the bug doesn't materialize with the change:
>
> [fix_spill_read_underrun.log](https://github.com/user-attachments/files/25119991/fix_spill_read_underrun.log) [main.log](https://github.com/user-attachments/files/25119993/main.log) [clickbench_partitioned.json - fix](https://github.com/user-attachments/files/25119997/clickbench_partitioned.json) [clickbench_partitioned.json - main](https://github.com/user-attachments/files/25120001/clickbench_partitioned.json)
Here's the comparison:
```
βββββββββββββ³βββββββββββββ³βββββββββββββ³ββββββββββββββββ
β Query β spill-main β spill β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.42 ms β 1.29 ms β +1.10x faster β
β QQuery 1 β 12.26 ms β 10.22 ms β +1.20x faster β
β QQuery 2 β 24.56 ms β 21.90 ms β +1.12x faster β
β QQuery 3 β 54.30 ms β 52.67 ms β no change β
β QQuery 4 β 589.73 ms β 544.75 ms β +1.08x faster β
β QQuery 5 β FAIL β FAIL β incomparable β
β QQuery 6 β 22.91 ms β 8.33 ms β +2.75x faster β
β QQuery 7 β 13.95 ms β 38.71 ms β 2.78x slower β
β QQuery 8 β 1100.09 ms β 1089.94 ms β no change β
β QQuery 9 β 476.05 ms β 443.06 ms β +1.07x faster β
β QQuery 10 β 124.00 ms β 115.41 ms β +1.07x faster β
β QQuery 11 β 134.52 ms β 122.89 ms β +1.09x faster β
β QQuery 12 β FAIL β FAIL β incomparable β
β QQuery 13 β FAIL β FAIL β incomparable β
β QQuery 14 β FAIL β FAIL β incomparable β
β QQuery 15 β 1013.17 ms β 1038.87 ms β no change β
β QQuery 16 β FAIL β FAIL β incomparable β
β QQuery 17 β FAIL β FAIL β incomparable β
β QQuery 18 β FAIL β FAIL β incomparable β
β QQuery 19 β 136.08 ms β 88.19 ms β +1.54x faster β
β QQuery 20 β 439.50 ms β 1187.21 ms β 2.70x slower β
β QQuery 21 β 608.01 ms β 765.89 ms β 1.26x slower β
β QQuery 22 β 1248.89 ms β 1256.74 ms β no change β
β QQuery 23 β 6601.82 ms β 4627.75 ms β +1.43x faster β
β QQuery 24 β 68.26 ms β 78.68 ms β 1.15x slower β
β QQuery 25 β 121.97 ms β 100.81 ms β +1.21x faster β
β QQuery 26 β 85.14 ms β 77.47 ms β +1.10x faster β
β QQuery 27 β 615.13 ms β 524.61 ms β +1.17x faster β
β QQuery 28 β FAIL β FAIL β incomparable β
β QQuery 29 β 223.85 ms β 201.93 ms β +1.11x faster β
β QQuery 30 β 539.42 ms β 495.41 ms β +1.09x faster β
β QQuery 31 β 1123.87 ms β 1016.04 ms β +1.11x faster β
β QQuery 32 β 5227.16 ms β 5989.61 ms β 1.15x slower β
β QQuery 33 β FAIL β FAIL β incomparable β
β QQuery 34 β FAIL β FAIL β incomparable β
β QQuery 35 β 999.16 ms β 991.90 ms β no change β
β QQuery 36 β FAIL β 365.41 ms β incomparable β
β QQuery 37 β 37.26 ms β 38.01 ms β no change β
β QQuery 38 β 58.04 ms β 59.82 ms β no change β
β QQuery 39 β 297.38 ms β 562.74 ms β 1.89x slower β
β QQuery 40 β 19.25 ms β 20.12 ms β no change β
β QQuery 41 β 18.30 ms β 17.35 ms β +1.05x faster β
β QQuery 42 β 16.78 ms β 14.57 ms β +1.15x faster β
βββββββββββββ΄βββββββββββββ΄βββββββββββββ΄ββββββββββββββββ
βββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββ©
β Total Time (spill-main) β 22052.23ms β
β Total Time (spill) β 21602.90ms β
β Average Time (spill-main) β 689.13ms β
β Average Time (spill) β 675.09ms β
β Queries Faster β 18 β
β Queries Slower β 6 β
β Queries with No Change β 8 β
β Queries with Failure β 11 β
βββββββββββββββββββββββββββββ΄βββββββββββββ
````
This looks good to me, there's some slowdowns but overwhelmingly speedups and fails -> not fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3856853745
I got it to finish!
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ./target/profiling/datafusion-cli -m 1G -c "SELECT \"UserID\", extract(minute FROM to_timestamp_seconds(\"EventTime\")) AS m, \"SearchPhrase\", COUNT(*) FROM '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned' GROUP BY \"UserID\", m, \"SearchPhrase\" ORDER BY COUNT(*) DESC LIMIT 10;"
DataFusion CLI v52.1.0
+---------------------+----+--------------+----------+
| UserID | m | SearchPhrase | count(*) |
+---------------------+----+--------------+----------+
| 1313338681122956954 | 31 | | 589 |
| 1313338681122956954 | 28 | | 578 |
| 1313338681122956954 | 29 | | 572 |
| 1313338681122956954 | 33 | | 567 |
| 1313338681122956954 | 27 | | 557 |
| 1313338681122956954 | 32 | | 554 |
| 1313338681122956954 | 30 | | 552 |
| 1313338681122956954 | 34 | | 546 |
| 1313338681122956954 | 26 | | 540 |
| 1313338681122956954 | 10 | | 539 |
+---------------------+----+--------------+----------+
10 row(s) fetched.
Elapsed 233.198 seconds.
```
That is a crazy long time though
For comparison, without a memory limit it takes 1s
```
andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ./target/profiling/datafusion-cli -c "SELECT \"UserID\", extract(minute FROM to_timestamp_seconds(\"EventTime\")) AS m, \"SearchPhrase\", COUNT(*) FROM '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned' GROUP BY \"UserID\", m, \"SearchPhrase\" ORDER BY COUNT(*) DESC LIMIT 10;"
DataFusion CLI v52.1.0
+---------------------+----+--------------+----------+
| UserID | m | SearchPhrase | count(*) |
+---------------------+----+--------------+----------+
| 1313338681122956954 | 31 | | 589 |
| 1313338681122956954 | 28 | | 578 |
| 1313338681122956954 | 29 | | 572 |
| 1313338681122956954 | 33 | | 567 |
| 1313338681122956954 | 27 | | 557 |
| 1313338681122956954 | 32 | | 554 |
| 1313338681122956954 | 30 | | 552 |
| 1313338681122956954 | 34 | | 546 |
| 1313338681122956954 | 26 | | 540 |
| 1313338681122956954 | 10 | | 539 |
+---------------------+----+--------------+----------+
10 row(s) fetched.
Elapsed 1.330 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#discussion_r2771477998
##########
datafusion/physical-plan/src/spill/spill_pool.rs:
##########
@@ -194,6 +194,8 @@ impl SpillPoolWriter {
// Append the batch
if let Some(ref mut writer) = file_shared.writer {
writer.append_batch(batch)?;
+ // make sure we flush the writer for readers
Review Comment:
I do wonder if it is ok to flush each batch (will this result in too many small file IOs?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3886888226
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and fix_spill_read_underrun
--------------------
Benchmark tpch_mem_sf1.json
--------------------
βββββββββββββ³ββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β fix_spill_read_underrun β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 100.96 ms β 100.65 ms β no change β
β QQuery 2 β 30.95 ms β 32.23 ms β no change β
β QQuery 3 β 40.03 ms β 40.42 ms β no change β
β QQuery 4 β 30.77 ms β 29.96 ms β no change β
β QQuery 5 β 91.37 ms β 88.91 ms β no change β
β QQuery 6 β 20.83 ms β 20.77 ms β no change β
β QQuery 7 β 157.18 ms β 167.99 ms β 1.07x slower β
β QQuery 8 β 40.60 ms β 39.05 ms β no change β
β QQuery 9 β 95.52 ms β 105.55 ms β 1.11x slower β
β QQuery 10 β 67.34 ms β 66.79 ms β no change β
β QQuery 11 β 18.71 ms β 18.85 ms β no change β
β QQuery 12 β 51.53 ms β 52.18 ms β no change β
β QQuery 13 β 49.24 ms β 48.37 ms β no change β
β QQuery 14 β 15.18 ms β 14.73 ms β no change β
β QQuery 15 β 29.55 ms β 30.08 ms β no change β
β QQuery 16 β 28.73 ms β 28.18 ms β no change β
β QQuery 17 β 140.76 ms β 140.44 ms β no change β
β QQuery 18 β 278.74 ms β 277.55 ms β no change β
β QQuery 19 β 40.02 ms β 40.27 ms β no change β
β QQuery 20 β 55.10 ms β 54.61 ms β no change β
β QQuery 21 β 189.74 ms β 189.32 ms β no change β
β QQuery 22 β 22.13 ms β 22.40 ms β no change β
βββββββββββββ΄ββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1594.98ms β
β Total Time (fix_spill_read_underrun) β 1609.29ms β
β Average Time (HEAD) β 72.50ms β
β Average Time (fix_spill_read_underrun) β 73.15ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 20 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3991936336
> @cetra3 @adriangb @alamb : Is there any interest in a 52.3.0 release? This bug is causing some problems for us. I am happy to create an issue/backports etc.
The 53.0.0 release is going to be made soon - could you upgrade to that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3856801501
I tried the reproducer and now it fails like this
```
./target/profiling/datafusion-cli -m 1G -c "SELECT \"UserID\", extract(minute FROM to_timestamp_seconds(\"EventTime\")) AS m, \"SearchPhrase\", COUNT(*) FROM '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned' GROUP BY \"UserID\", m, \"SearchPhrase\" ORDER BY COUNT(*) DESC LIMIT 10;"
DataFusion CLI v52.1.0
Error: Execution error: (Hint: you may increase the file descriptor limit with shell command 'ulimit -n 4096') Failed to create partition file at "/var/folders/1l/tg68jc6550gg8xqf1hr4mlwr0000gn/T/.tmp2YLiiK/.tmp8w9Wx2": Os { code: 24, kind: Uncategorized, message: "Too many open files" }
```
However, when I cranked the ulimit via
```shell
ulimit -n 10000
```
Then it fails like this
```
3:~/Software/datafusion$ ./target/profiling/datafusion-cli -m 1G -c "SELECT \"UserID\", extract(minute FROM to_timestamp_seconds(\"EventTime\")) AS m, \"SearchPhrase\", COUNT(*) FROM '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned' GROUP BY \"UserID\", m, \"SearchPhrase\" ORDER BY COUNT(*) DESC LIMIT 10;"
DataFusion CLI v52.1.0
Error: Resources exhausted: Additional allocation failed for TopK[7] with top memory consumers (across reservations) as:
GroupedHashAggregateStream[7] (count(1))#48(can spill: true) consumed 83.8 MB, peak 83.8 MB,
GroupedHashAggregateStream[4] (count(1))#26(can spill: true) consumed 83.8 MB, peak 83.8 MB,
GroupedHashAggregateStream[15] (count(1))#64(can spill: true) consumed 79.1 MB, peak 79.1 MB.
Error: Failed to allocate additional 2.3 MB for TopK[7] with 7.4 MB already allocated for this reservation - 334.3 KB remain available for the total pool
```
That being said, this seems like an improvement to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3885944201
π€ Hi @adriangb, thanks for the request (https://github.com/apache/datafusion/pull/20159#issuecomment-3885943088).
[`scrape_comments.py`](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/scrape_comments.py) only supports whitelisted benchmarks.
- Standard: clickbench_1, clickbench_extended, clickbench_partitioned, clickbench_pushdown, external_aggr, tpcds, tpch, tpch10, tpch_mem, tpch_mem10
- Criterion: aggregate_query_sql, aggregate_vectorized, case_when, character_length, in_list, left, range_and_generate_series, reset_plan_states, sort, sql_planner, strpos, substr_index, with_hashes
Please choose one or more of these with `run benchmark <name>` or `run benchmark <name1> <name2>...`
You can also set environment variables on subsequent lines:
```
run benchmark tpch_mem
DATAFUSION_RUNTIME_MEMORY_LIMIT=1G
```
Unsupported benchmarks: clickbench.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "hareshkh (via GitHub)" <gi...@apache.org>.
hareshkh commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3993785198
@adriangb - Thankyou!
I have created a issue here - https://github.com/apache/datafusion/issues/20681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3887411270
π€: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and fix_spill_read_underrun
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β fix_spill_read_underrun β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 2.53 ms β 2.63 ms β no change β
β QQuery 1 β 50.33 ms β 52.74 ms β no change β
β QQuery 2 β 131.69 ms β 132.69 ms β no change β
β QQuery 3 β 149.89 ms β 140.35 ms β +1.07x faster β
β QQuery 4 β 954.08 ms β 972.39 ms β no change β
β QQuery 5 β 1265.48 ms β 1245.69 ms β no change β
β QQuery 6 β 6.10 ms β 6.75 ms β 1.11x slower β
β QQuery 7 β 52.54 ms β 56.16 ms β 1.07x slower β
β QQuery 8 β 1322.73 ms β 1381.15 ms β no change β
β QQuery 9 β 1694.60 ms β 1764.09 ms β no change β
β QQuery 10 β 341.60 ms β 349.58 ms β no change β
β QQuery 11 β 387.24 ms β 398.72 ms β no change β
β QQuery 12 β 1167.39 ms β 1208.82 ms β no change β
β QQuery 13 β 1805.52 ms β 1871.47 ms β no change β
β QQuery 14 β 1178.40 ms β 1239.09 ms β 1.05x slower β
β QQuery 15 β 1139.39 ms β 1162.29 ms β no change β
β QQuery 16 β 2352.37 ms β 2424.55 ms β no change β
β QQuery 17 β 2365.02 ms β 2399.04 ms β no change β
β QQuery 18 β 5693.57 ms β 4904.76 ms β +1.16x faster β
β QQuery 19 β 124.37 ms β 121.15 ms β no change β
β QQuery 20 β 2050.69 ms β 1857.09 ms β +1.10x faster β
β QQuery 21 β 2195.80 ms β 2168.18 ms β no change β
β QQuery 22 β 3817.96 ms β 3615.36 ms β +1.06x faster β
β QQuery 23 β 12205.92 ms β 11924.73 ms β no change β
β QQuery 24 β 210.40 ms β 210.83 ms β no change β
β QQuery 25 β 476.44 ms β 449.48 ms β +1.06x faster β
β QQuery 26 β 206.18 ms β 208.65 ms β no change β
β QQuery 27 β 2653.55 ms β 2598.37 ms β no change β
β QQuery 28 β 23224.25 ms β 24275.43 ms β no change β
β QQuery 29 β 978.23 ms β 977.71 ms β no change β
β QQuery 30 β 1218.98 ms β 1243.37 ms β no change β
β QQuery 31 β 1297.23 ms β 1275.84 ms β no change β
β QQuery 32 β 4226.60 ms β 4407.76 ms β no change β
β QQuery 33 β 5666.18 ms β 5336.43 ms β +1.06x faster β
β QQuery 34 β 5824.31 ms β 5811.97 ms β no change β
β QQuery 35 β 1806.25 ms β 1845.91 ms β no change β
β QQuery 36 β 192.53 ms β 189.65 ms β no change β
β QQuery 37 β 71.69 ms β 72.55 ms β no change β
β QQuery 38 β 113.89 ms β 112.62 ms β no change β
β QQuery 39 β 346.96 ms β 336.28 ms β no change β
β QQuery 40 β 37.70 ms β 38.82 ms β no change β
β QQuery 41 β 33.55 ms β 34.75 ms β no change β
β QQuery 42 β 31.25 ms β 31.01 ms β no change β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 91071.42ms β
β Total Time (fix_spill_read_underrun) β 90856.88ms β
β Average Time (HEAD) β 2117.94ms β
β Average Time (fix_spill_read_underrun) β 2112.95ms β
β Queries Faster β 6 β
β Queries Slower β 3 β
β Queries with No Change β 34 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3886772627
π€ `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing fix_spill_read_underrun (572cd33b9781b74ec698149d9f81f3c13904561d) to 8ba3d2617b49afa7473e0b583acbb2bf93928523 [diff](https://github.com/apache/datafusion/compare/8ba3d2617b49afa7473e0b583acbb2bf93928523..572cd33b9781b74ec698149d9f81f3c13904561d) using: tpch_mem
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "hareshkh (via GitHub)" <gi...@apache.org>.
hareshkh commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3991859852
@cetra3 @adriangb @alamb : Is there any interest in a 52.3.0 release? This bug is causing some problems for us. I am happy to create an issue/backports etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "juice928 (via GitHub)" <gi...@apache.org>.
juice928 commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3857499530
π Hi, I'm an automated AI code review bot. I ran some checks on this PR and found **2 points that might be worth attention** (could be false positives, please use your judgment):
1. <strong>Optimizing the frequency of disk flushing during spill writes</strong>
- [Location](https://github.com/apache/datafusion/blob/572cd33b9781b74ec698149d9f81f3c13904561d/datafusion/physical-plan/src/spill/spill_pool.rs#L197-L198): `datafusion/physical-plan/src/spill/spill_pool.rs:L197-L198`
- Impact: Frequent manual flushing may bypass the benefits of buffering and lead to performance overhead in spill-heavy workloads.
- Suggestion: Consider removing the explicit `flush()` call to allow the internal buffering of the writer to handle efficiency more naturally.
2. <strong>Improving the synchronization between concurrent spill file reads and writes</strong>
- [Location](https://github.com/apache/datafusion/blob/572cd33b9781b74ec698149d9f81f3c13904561d/datafusion/physical-plan/src/spill/spill_pool.rs#L540-L544): `datafusion/physical-plan/src/spill/spill_pool.rs:L540-L544`
- Impact: A reader might encounter an early EOF and terminate before the writer finishes, potentially leading to incomplete data processing.
- Suggestion: Consider implementing a "tailing" mechanism for the reader or ensuring the reader starts only after the writer has finalized the data segment.
If you find these suggestions disruptive, you can reply "stop" , and I'll automatically skip this repository in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #20159:
URL: https://github.com/apache/datafusion/pull/20159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#discussion_r2771510814
##########
datafusion/physical-plan/src/spill/spill_pool.rs:
##########
@@ -194,6 +194,8 @@ impl SpillPoolWriter {
// Append the batch
if let Some(ref mut writer) = file_shared.writer {
writer.append_batch(batch)?;
+ // make sure we flush the writer for readers
Review Comment:
We need to do this, or we need to ensure data is written before we try reading it, maybe we could do it periodically rather than each batch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3884368412
> @alamb Wdyt of adding a benchmark that forces spilling? Or an option to the current script to run with a memory limit?
https://github.com/alamb/datafusion-benchmarking/pull/11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3885730529
Seems to have done the trick
```shell
# Automatically created by scrape_comments.py
# PR: https://github.com/apache/datafusion/pull/20159
# Comment: https://github.com/apache/datafusion/pull/20159#issuecomment-3885722084
# User: alamb
# Body: run benchmark tpch_mem^M
DATAFUSION_RUNTIME_MEMORY_LIMIT=1G
#!/usr/bin/env bash
set -euo pipefail
PR_URL="https://github.com/apache/datafusion/pull/20159"
OUTPUT_FILE="/tmp/benchmark_script_output.txt"
error_handler() {
local exit_code=$?
set +e
local tail_output
tail_output="$(tail -n 10 "$OUTPUT_FILE" 2>/dev/null || true)"
local body_file
body_file="$(mktemp)"
{
echo "Benchmark script failed with exit code ${exit_code}."
echo
echo "Last 10 lines of output:"
echo "<details><summary>Click to expand</summary>"
echo
echo '```'
echo "${tail_output}"
echo '```'
echo
echo "</details>"
} > "${body_file}"
gh pr comment "${PR_URL}" --body-file "${body_file}"
rm -f "${body_file}"
exit 0
trap error_handler ERR
: > "${OUTPUT_FILE}"
exec > >(tee -a "${OUTPUT_FILE}") 2>&1
export DATAFUSION_RUNTIME_MEMORY_LIMIT=1G
BENCHMARKS="tpch_mem" ./gh_compare_branch.sh https://github.com/apache/datafusion/pull/20159
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3885723007
Let's run it and see what happens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3887339882
π€ `./gh_compare_branch.sh` [gh_compare_branch.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_branch.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing fix_spill_read_underrun (572cd33b9781b74ec698149d9f81f3c13904561d) to 8ba3d2617b49afa7473e0b583acbb2bf93928523 [diff](https://github.com/apache/datafusion/compare/8ba3d2617b49afa7473e0b583acbb2bf93928523..572cd33b9781b74ec698149d9f81f3c13904561d) using: clickbench_partitioned
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3885945509
run benchmark clickbench_partitioned
DATAFUSION_RUNTIME_MEMORY_LIMIT=500M
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3857764965
OK I have tried to get some benchmark results out using disk spilling and the `tpch` benchmark. Setting the memory limit to 1G I can successfully run this on this branch all the time. On main it fails a lot with the arrow error but managed to get it to work once or twice. However it just looks like it's `Query 21` where any spilling happens.
On this branch:
```
Query 21 iteration 0 took 1596.5 ms and returned 100 rows
Query 21 iteration 1 took 2269.5 ms and returned 100 rows
Query 21 iteration 2 took 2515.2 ms and returned 100 rows
Query 21 iteration 3 took 2907.9 ms and returned 100 rows
Query 21 iteration 4 took 3504.3 ms and returned 100 rows
Query 21 avg time: 2558.70 ms
```
On `main` the best I got was:
```
Query 21 iteration 0 took 3121.8 ms and returned 100 rows
Query 21 iteration 1 took 2731.7 ms and returned 100 rows
Query 21 iteration 2 took 3314.0 ms and returned 100 rows
Query 21 iteration 3 took 4073.1 ms and returned 100 rows
Query 21 iteration 4 took 5101.1 ms and returned 100 rows
Query 21 avg time: 3668.34 ms
```
I've attached the JSON of both runs:
[main.json](https://github.com/user-attachments/files/25117450/main.json)
[fix_spill_read_underrun.json](https://github.com/user-attachments/files/25117448/fix_spill_read_underrun.json)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Fix Arrow Spill Underrun [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #20159:
URL: https://github.com/apache/datafusion/pull/20159#issuecomment-3858244351
> I haven't written a test, but I have been running the example in the attached issue. While it now fails with allocation errors, the original error goes away.
Call for a test, might extract from the example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org