[XL] Change in asterixdb[master]: working on reading shredding

[AsterixDB Code Review <do...@asterix-gerrit.ics.uci.edu>]

From Hussain Towaileb <hu...@gmail.com>:

Hussain Towaileb has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21536?usp=email )


Change subject: working on reading shredding
......................................................................

working on reading shredding

Details:

Adds two independent read optimizations for Iceberg shredded
VARIANT columns. Projection pushdown reads only the shredded
sub-columns a query references; predicate pushdown skips whole
data files using the per-sub-field bounds in the manifest. The
two are orthogonal and gated separately.

Projection pushdown derives the requested sub-paths from the
projected record type the optimizer already produces, clips the
Parquet typed_value group to those members, and reads the file
with the clipped schema. Iceberg's own reader cannot express
this: a variant is a single opaque type in the Iceberg schema,
so ReadConf always requests every typed_value sub-column. The
metadata and residual value columns are always kept. Anything
that cannot be narrowed, and any failure, falls back to the
standard read path. Controlled by variant-projection-pushdown,
default on.

Predicate pushdown rewrites the dotted sub-field reference the
filter builder emits into an Iceberg extract term, the only form
that binds against a variant. It runs in the reader factory, the
first point where the Iceberg schema is known. A predicate that
cannot be rewritten is left untouched, never dropped.

Important: Iceberg 1.11.0 cannot carry an extract term through
TableScan. ExpressionUtil.describe throws on one, and
SnapshotScan.planFiles sanitizes the filter unconditionally, so
planning fails outright. It also reads variant manifest bounds
with the wrong byte order. Until
https://github.com/apache/iceberg/pull/15384 is merged and picked
up in the Iceberg release we depend on, only the extract-free part
of the filter is pushed and the bounds are evaluated locally.
Every workaround site is marked iceberg-15384, a tripwire test
fails on upgrade, and the removal checklist lives on
VariantBoundsEvaluator.

A data file is skipped only when its bounds prove that no row can
match. Bounds that are absent, unparseable or of a type we cannot
compare keep the file, as do NOT and !=. Temporal, binary and
UUID sub-fields are never compared, because they arrive as bare
numbers that would compare successfully against the wrong thing.

Projection pushdown is skipped for scan tasks that carry deletes,
where the read is materialized against a widened schema. Predicate
pushdown is unaffected, since deletes only ever remove rows. No
code here is delete-kind aware, so equality deletes and deletion
vectors both work through Iceberg's own delete filter.

Added tests for requested-path extraction, typed_value clipping,
VARIANT gating, predicate rewriting, the scan workaround, bound
evaluation over every variant physical type, sub-field pushdown
per type, equality deletes, deletion vectors, and multi-row-group
files.

Ext-ref: MB-73189
Change-Id: I719673088db0fc809ec9ed8b208ae418690bcdbb
---
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/iceberg/IcebergTest.java
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletes/test.000.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletes/test.010.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletes/test.030.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletes/test.040.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletes/test.050.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletes/test.999.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletion-vectors/test.000.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletion-vectors/test.010.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletion-vectors/test.030.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletion-vectors/test.040.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletion-vectors/test.050.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-deletion-vectors/test.999.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-row-groups/test.000.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-row-groups/test.010.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-row-groups/test.030.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-row-groups/test.040.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-row-groups/test.999.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.000.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.010.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.030.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.040.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.050.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.060.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.070.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.080.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.090.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/iceberg/variant-stats-pushdown/test.999.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-deletes/result.030.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-deletes/result.040.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-deletes/result.050.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-deletion-vectors/result.030.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-deletion-vectors/result.040.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-deletion-vectors/result.050.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-row-groups/result.030.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-row-groups/result.040.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.030.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.040.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.050.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.060.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.070.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.080.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/iceberg/variant-stats-pushdown/result.090.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_iceberg.xml
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/iceberg/IcebergFileRecordReader.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/iceberg/IcebergParquetRecordReaderFactory.java
A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/aws/iceberg/VariantProjectedParquetReader.java
M asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/iceberg/RequestedVariantPaths.java
A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/iceberg/VariantBoundsEvaluator.java
A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/iceberg/VariantPredicateRewriter.java
A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/iceberg/VariantProjectionPlan.java
A asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/iceberg/VariantSchemaClipper.java
A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/util/iceberg/ReadingShreddedPushdownTest.java
A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/util/iceberg/VariantBoundsAllTypesTest.java
A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/util/iceberg/VariantBoundsEvaluatorTest.java
56 files changed, 4,531 insertions(+), 3 deletions(-)



  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/36/21536/1


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21536?usp=email
To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email

Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Change-Id: I719673088db0fc809ec9ed8b208ae418690bcdbb
Gerrit-Change-Number: 21536
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hu...@gmail.com>