[I] `binary_as_string` over a Parquet `Binary` column with invalid UTF-8 causes a segfault [datafusion]

["adriangb (via GitHub)" <gi...@apache.org>]

adriangb opened a new issue, #25509:
URL: https://github.com/apache/datafusion/issues/25509

   **Describe the bug**
   
   `datafusion.execution.parquet.binary_as_string = true` changes a Parquet `Binary` column to a string type in the schema that DataFusion gives to the Parquet reader. If the column holds bytes that are not valid UTF-8, the reader returns an invalid string array and reports no error. The first operation that decodes those bytes then stops the process with SIGSEGV. The user gets no message and no error code from the query.
   
   This is not limited to unusual schemas. A top level column is sufficient.
   
   **To Reproduce**
   
   ```shell
   datafusion-cli -c "COPY (SELECT decode('ff','hex') AS b) TO 'bin.parquet'" -c "set datafusion.execution.parquet.binary_as_string = true" -c "SELECT b FROM 'bin.parquet'"
   ```
   
   ```
   DataFusion CLI v54.0.0
   +-------+
   | count |
   +-------+
   | 1     |
   +-------+
   1 row(s) fetched.
   Elapsed 0.010 seconds.
   
   0 row(s) fetched.
   Elapsed 0.001 seconds.
   ```
   
   The exit code is 139. The macOS crash report shows `SIGSEGV` with the fault in `comfy_table::Row::max_content_widths`, which measures the width of the invalid string. With `SELECT upper(b)` the fault moves to `alloc::str::to_uppercase`.
   
   Measured on the released `datafusion-cli` 54.0.0 and on a debug build of `main` at 4e907557ad (arrow and parquet 59.3.0):
   
   | Column type and target | Build | Result |
   | --- | --- | --- |
   | `Binary` to `Utf8View` (defaults) | release | SIGSEGV |
   | `Binary` to `Utf8View` (defaults) | debug | non-unwinding panic from the `core::str::validations` UB check, then abort |
   | `Binary` to `Utf8` (`schema_force_view_types = false`) | release | SIGSEGV |
   | `Binary` to `Utf8` | debug | panic in `parquet-59.3.0/src/arrow/buffer/offset_buffer.rs:141`: `Invalid UTF8 sequence at string index 0` |
   | `LargeBinary` to `LargeUtf8` | release | SIGSEGV |
   | `BinaryView` to `Utf8View` | release | SIGSEGV |
   
   `count(*)`, `octet_length(b)` and the byte-wise comparison `b = 'ok'` all succeed, because they do not decode UTF-8. The invalid array moves through the plan without a sign of a problem.
   
   An explicit cast is safe, which shows the difference:
   
   ```shell
   datafusion-cli -c "COPY (SELECT decode('ff','hex') AS b) TO 'bin.parquet'" -c "SELECT arrow_cast(b, 'Utf8') FROM 'bin.parquet'"
   ```
   
   ```
   Error: Arrow error: Invalid argument error: Encountered non UTF-8 data: invalid utf-8 sequence of 1 bytes from index 0
   ```
   
   Valid multibyte UTF-8 in a `Binary` column reads correctly with the option on.
   
   **Expected behavior**
   
   The query must fail with an error, in the same way as the explicit cast. DataFusion must not give an invalid string array to the rest of the plan.
   
   **Additional context**
   
   The cause is in parquet-rs. `ByteArrayColumnValueDecoder::new` selects UTF-8 validation from the physical Parquet annotation only:
   
   ```rust
   // parquet/src/arrow/array_reader/byte_array.rs:192
   let validate_utf8 = desc.converted_type() == ConvertedType::UTF8;
   ```
   
   A plain BYTE_ARRAY column has no `UTF8` annotation, so the target Arrow type that DataFusion supplies through `ArrowReaderOptions::with_schema` has no effect. `OffsetBuffer::into_array` then uses `ArrayDataBuilder::build_unchecked` in release builds. Upstream report: PLACEHOLDER_ARROW
   
   Found while reviewing https://github.com/apache/datafusion/pull/25342. That PR does not introduce the problem. It applies the same override to map children, so it makes more reads reach this path.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org