You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/08/14 14:39:47 UTC

[PR] Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) [datafusion]

alamb opened a new pull request, #24373:
URL: https://github.com/apache/datafusion/pull/24373

   ## Which issue does this PR close?
   
   - Part of testing https://github.com/apache/arrow-rs/pull/9372
   
   ## Rationale for this change
   
   Test PR (not for merge) to try out the Parquet ALP encoder/decoder support proposed in https://github.com/apache/arrow-rs/pull/9372 from DataFusion, so ALP encoded parquet files can be written and read with `datafusion-cli`.
   
   Based on https://github.com/apache/datafusion/pull/24366 (update to arrow-rs main), with the `[patch.crates-io]` section repointed at the `alp` branch from https://github.com/sdf-jkl/arrow-rs.
   
   ## What changes are included in this PR?
   
   - Pin the arrow/parquet git patch to `sdf-jkl/arrow-rs` branch `alp` (currently commit `5e9ad1c6`)
   - Adjust `try_fb_to_schema` back to `fb_to_schema` (the `alp` branch is based on an arrow-rs main commit that predates that API)
   - Add `alp` as a recognized value for the parquet `encoding` writer option
   
   ## Are these changes tested?
   
   Manually, with `datafusion-cli`:
   
   ```sql
   > COPY (SELECT random() AS r, CAST(random() AS FLOAT) AS f FROM generate_series(1,10000))
     TO '/tmp/alp_test.parquet'
     OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' 'false');
   
   > SELECT path_in_schema, type, encodings FROM parquet_metadata('/tmp/alp_test.parquet');
   +----------------+--------+------------+
   | path_in_schema | type   | encodings  |
   +----------------+--------+------------+
   | "r"            | DOUBLE | [RLE, ALP] |
   | "f"            | FLOAT  | [RLE, ALP] |
   +----------------+--------+------------+
   
   > SELECT count(*), min(r) >= 0, max(r) <= 1, min(f) >= 0 FROM '/tmp/alp_test.parquet';
   -- 10000, true, true, true
   ```
   
   Only `datafusion-cli` is expected to compile/run; other tests are not expected to pass.
   
   ## Are there any user-facing changes?
   
   No (test PR, not intended for merge).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #24373: Test: read/write ALP encoded parquet files (apache/arrow-rs#9372)
URL: https://github.com/apache/datafusion/pull/24373


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Test: read/write ALP encoded parquet files (apache/arrow-rs#9372) [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #24373:
URL: https://github.com/apache/datafusion/pull/24373#issuecomment-5604594472

   Upstream PR is merged


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org