[jira] [Created] (SPARK-59853) Accept a short last bit-packed run in the vectorized RLE reader

["L. C. Hsieh (Jira)" <ji...@apache.org>]

L. C. Hsieh created SPARK-59853:
-----------------------------------

             Summary: Accept a short last bit-packed run in the vectorized RLE reader
                 Key: SPARK-59853
                 URL: https://issues.apache.org/jira/browse/SPARK-59853
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


The Parquet spec requires bit-packed runs in the RLE/bit-packing hybrid encoding to be padded to a multiple of 8 values, but some non-compliant writers write a last bit-packed run that is shorter than its padding, e.g. Polars before https://github.com/pola-rs/polars/pull/13883 (see https://github.com/pola-rs/polars/issues/13818, "Spark can't read parquet files written by polars").

The vectorized reader (VectorizedRleValuesReader) rejects such a run, even when its bytes cover every value that is read, while the non-vectorized reader reads it: parquet-java's RunLengthBitPackingHybridDecoder reads min(numGroups * bitWidth, available) bytes. Arrow C++ also accepts it since https://github.com/apache/arrow/pull/47992.

The vectorized reader could accept it the way Arrow does: only for the last run of the stream, and only the values fully covered by the remaining bytes, so reading past them still fails.

This came up in the review of SPARK-59832 (https://github.com/apache/spark/pull/59117), which keeps the existing strict behavior.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org