You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/03/27 21:57:30 UTC

[I] Shuffle memory pool accounting: peak RSS significantly exceeds configured memory limit [datafusion-comet]

andygrove opened a new issue, #3821:
URL: https://github.com/apache/datafusion-comet/issues/3821

   ## Description
   
   When running the shuffle benchmark with a configured memory pool limit, the actual peak RSS (resident set size) significantly exceeds the configured limit. This suggests there are significant memory allocations happening outside of the memory pool accounting.
   
   ## Benchmark Data
   
   **Setup:** TPCH SF100 lineitem (100M rows, 16 columns), hash partitioning (200 partitions), lz4 compression, single iteration.
   
   | Memory Limit | Peak RSS | RSS / Limit | Write Time | Throughput |
   |---|---|---|---|---|
   | 2 GB | 3.8 GB | 1.78x | 43.4s | 2.30M rows/s |
   | 4 GB | 6.7 GB | 1.56x | 45.1s | 2.22M rows/s |
   | 8 GB | 11.0 GB | 1.28x | 46.4s | 2.15M rows/s |
   | 16 GB | 13.1 GB | 0.76x | 51.9s | 1.93M rows/s |
   
   ## Observations
   
   1. **Peak RSS exceeds the memory limit by up to 1.78x.** At a 2 GB memory limit, the process uses 3.8 GB of RSS. This means nearly half the memory usage is untracked by the memory pool.
   
   2. **Overhead outside the pool grows with the limit.** Subtracting the memory limit from peak RSS: 2g has ~1.8 GB overhead, 4g has ~2.7 GB, 8g has ~3.0 GB. This suggests some internal structures scale with available memory rather than being fixed-size.
   
   3. **Higher memory limits are slower, not faster.** The 2 GB run (43.4s) is 20% faster than the 16 GB run (51.9s). Smaller pools may keep the working set more cache-friendly, while larger buffers may cause more memory pressure at flush time.
   
   ## Expected Behavior
   
   The memory pool limit should more closely bound the total process memory usage. Untracked allocations (Arrow buffers, Parquet reader state, partition writer buffers, etc.) should either be accounted for in the pool or documented as expected overhead.
   
   ## Reproduction
   
   ```sh
   cargo build --release --bin shuffle_bench --features shuffle-bench
   
   /usr/bin/time -l ./native/target/release/shuffle_bench \
       --input /opt/tpch/sf100/lineitem \
       --codec lz4 \
       --partitions 200 \
       --hash-columns 0,3 \
       --memory-limit 2147483648 \
       --limit 100000000
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Shuffle memory pool accounting: peak RSS significantly exceeds configured memory limit [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3821: Shuffle memory pool accounting: peak RSS significantly exceeds configured memory limit
URL: https://github.com/apache/datafusion-comet/issues/3821


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Shuffle memory pool accounting: peak RSS significantly exceeds configured memory limit [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3821:
URL: https://github.com/apache/datafusion-comet/issues/3821#issuecomment-4391560809

   In 0.15.0 release there was a fix to stop allocating 2x available memory for shuffles. I will close this for now. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Shuffle memory pool accounting: peak RSS significantly exceeds configured memory limit [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3821:
URL: https://github.com/apache/datafusion-comet/issues/3821#issuecomment-4236794752

   Root cause is likely 3921


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org