[I] External sort spill GC inflates deduplicated string views [datafusion]

["cetra3 (via GitHub)" <gi...@apache.org>]

cetra3 opened a new issue, #23564:
URL: https://github.com/apache/datafusion/issues/23564

   ### Describe the bug
   
   A recent PR adjusted the way the spill sort works by running `gc()` on string views: http://github.com/apache/datafusion/pull/21633
   
   While this did reduce the file size considerably in some cases, in other cases this has caused a *regression* in spill file sizes.  Namely, a highly deduplicated string view, will inflate considerably as `gc()` by [its contract *does not* deduplicate](https://docs.rs/arrow-array/latest/arrow_array/array/struct.GenericByteViewArray.html#method.gc):
   
   > Note: this function does not attempt to canonicalize / deduplicate values. For this feature see [GenericByteViewBuilder::with_deduplicate_strings](https://docs.rs/arrow-array/latest/arrow_array/builder/struct.GenericByteViewBuilder.html#method.with_deduplicate_strings).
   
   I.e, in a production workload I saw, trying to sort ~1gb of compressed parquet data *blew out* to **73gb** of spilled bytes....
   
   This is especially bad because we are spilling because we are already under memory pressure already, and reading *back* those inflated spilled files *increases* memory usage at the worse possible time.
   
   ### To Reproduce
   
   Create a deduped parquet file using datafusion-cli:
   
   ```sql
   COPY (
     SELECT value AS id,
            'container-id-' || lpad(CAST(value % 1000 AS VARCHAR), 50, '0') AS label
     FROM generate_series(1, 1000000)
   ) TO 'repro.parquet' STORED AS PARQUET;
   ```
   
   This creates a ~1.3mb parquet file.
   
   
   Sort it with a low memory limit and look at the explain output:
   
   ```bash
   datafusion-cli -m 64M -c "EXPLAIN ANALYZE SELECT * FROM 'repro.parquet' ORDER BY id;"
   ```
   
   You will see that the spilled bytes is significantly higher than the parquet file:
   
   ```
   DataSourceExec: ... bytes_scanned=1.37 M, output_bytes=31.0 MB
   SortExec: expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false],
     output_rows=1.00 M, output_bytes=108.8 MB,
     spill_count=2, spilled_bytes=83.2 MB, spilled_rows=1.00 M
   ```
   
   So in other words, **a 1.3 MB parquet file spills to 83.2 MB**
   
   ### Expected behavior
   
   Spilling does not allocate that much disk to spill a string view column.
   
   We should really be using something like:
   
   ```rust
     let mut builder = GenericByteViewBuilder::with_capacity(array.len())
         .with_deduplicate_strings();
     for v in array.iter() { builder.append_option(v); }
     builder.finish()
   ```
   
   With that in place, the spill bytes is much lower
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] External sort spill GC inflates deduplicated string views [datafusion]

["adriangb (via GitHub)" <gi...@apache.org>]

adriangb closed issue #23564: External sort spill GC inflates deduplicated string views
URL: https://github.com/apache/datafusion/issues/23564


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org