[jira] [Commented] (IMPALA-9821) Rewrite ds_hll_sketch() and ds_hll_union() and other datasketch generating functions to return Binary

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118942#comment-18118942 ] 

ASF subversion and git services commented on IMPALA-9821:
---------------------------------------------------------

Commit 893783e1190f924fa5bb097029ae5a84477690c8 in impala's branch refs/heads/master from Daniel Vanko
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=893783e11 ]

IMPALA-12675: Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by default and turn on validation

Now that Impala has a BINARY type for binary data and broader UTF-8
support, STRING columns written to Parquet are annotated with the
UTF-8 logical type by default (PARQUET_ANNOTATE_STRINGS_UTF8 now
defaults to true).

To keep the annotation truthful, STRING values are validated as UTF-8
on the Parquet write path (using the simdutf library) whenever the
annotation is enabled. A value that is not valid UTF-8 aborts the write
with PARQUET_INVALID_UTF8_STRING. Validation is an inseparable
consequence of the annotation: there is no option to keep the
annotation while writing unvalidated values, since that would produce a
file whose metadata lies about its contents. To store non-UTF-8 bytes,
set PARQUET_ANNOTATE_STRINGS_UTF8 to false to omit the annotation, or
use a BINARY column. CHAR/VARCHAR (always text) and BINARY columns are
not validated. Iceberg tables always annotate STRING columns as UTF-8
(a spec requirement that is not configurable), so their STRING values
are always validated; use a BINARY column to store non-UTF-8 bytes
there.

Details:
 * Consume the simdutf 9.0.0 library from the native toolchain:
   add IMPALA_SIMDUTF_VERSION, register the package in
   bin/bootstrap_toolchain.py, locate it with a new
   cmake_modules/FindSimdutf.cmake, and link it via IMPALA_DEPENDENCIES.
 * Validate once per non-null value in
   HdfsParquetTableWriter::AppendRow().
 * Returning error if the string is not valid UTF-8, showing the
   offending byte and its offset.
 * Update the docs, including a new Impala 5.0.x entry in
   impala_incompatible_changes.xml documenting the default flip as a
   behaviour change.

Validation adds little to write time. CTAS of tpch.lineitem (6M rows)
to Parquet, 30 runs each on a local 3-node minicluster release build
(trimmed mean):

    PARQUET_ANNOTATE_STRINGS_UTF8 | CTAS time
    ------------------------------+----------
    false (no validation)         | ~1.12s
    true  (validate)              | ~1.22s

That is about +0.1s, or ~9%. The SIMD-accelerated simdutf scan is cheap
relative to the rest of the write path.

Testing:
 * added new be test: Utf8UtilTest
 * set PARQUET_ANNOTATE_STRINGS_UTF8=false in test_datasketches.py so it
   can write the binary data into STRING columns (until IMPALA-9821)
 * add new tests to test_iceberg.py and test_scanners.py
 * modified existing e2e testcases to match new estimations

Change-Id: Ia95bcba978863ffb5d603fdf81bb2c75ba06f7da
Assisted-by: Claude Opus 4.8 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24570
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>


> Rewrite ds_hll_sketch() and ds_hll_union() and other datasketch generating functions to return Binary
> -----------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-9821
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9821
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend
>            Reporter: Gabor Kaszab
>            Assignee: Mihaly Szjatinya
>            Priority: Blocker
>
> Until Binary implementation is ongoing ds_hll_sketch() and ds_hll_union() functions return serialized sketches in String format. Once Binary is available in Impala these can return the serialized sketches in Binary format.
> Currently when sketches are written by Hive as BINARY to ORC table and this table is loaded to Impala where the sketch columns are STRINGs then we get an error
> {code:java}
> ERROR: Type mismatch: table column STRING is map to column binary in ORC file
> {code}
> Interestingly the works with Parquet format.
> Once we have binary support make sure to add coverage for ORC table where the table is created and populated by Hive and read for estimating by Impala.
> UPDATE: since BINARY type support was added in IMPALA-9482, the error above is not reproducable.
> Currently (2026 September) what remains is the incompatibility between datasketches function signatures, as Impala uses STRING in places where Hive used BINARY.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org