You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2026/10/19 10:02:47 UTC

(doris) branch master updated: [test](paimon) Cover Variant reads of a deduplicate primary-key Paimon table (#69227)

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new a7ee5ab2d68 [test](paimon) Cover Variant reads of a deduplicate primary-key Paimon table (#69227)
a7ee5ab2d68 is described below

commit a7ee5ab2d68bad37f2de6d4888be494ab07fafca
Author: Mingyu Chen (Rayner) <mo...@gmail.com>
AuthorDate: Mon Oct 19 11:02:36 2026 +0100

    [test](paimon) Cover Variant reads of a deduplicate primary-key Paimon table (#69227)
    
    ### What problem does this PR solve?
    
    Issue Number: None
    
    Related PR: #66466, #68242
    
    Problem Summary:
    
    **In short.** Ports the primary-key Variant coverage of branch-4.1
    #66466 to master: a Paimon deduplicate table whose second snapshot
    updates two keys, and three blocks of `test_paimon_catalog_variant` that
    read it through JNI. The rest of #66466 and the WAL fix #68242 are not
    ported, because master handles those cases another way; see below.
    
    **Background.** #66466 did three things on branch-4.x:
    
    1. In BE, a generic gate in `TableReader::validate_file_mapping` rejects
    a physical Variant read from any file format but Parquet. #68242 then
    exempted WAL, which stores Doris blocks and supports Variant, because
    the gate broke group commit WAL replay.
    2. In FE, `PaimonScanNode` rejects a query that projects a Variant
    column while Variant V2 is disabled.
    3. In the tests, `run13.sql` gets `variant_primary_key_smoke`, and
    `test_paimon_catalog_variant` reads it.
    
    **The problem, and what it cost.** Every Variant table master's
    `test_paimon_catalog_variant` reads is append-only, so the regression
    does not check that a Paimon merge read keeps only the latest row of
    each key while it materializes Variant values.
    
    **How this PR fixes it.**
    
    - `run13.sql` creates `variant_primary_key_smoke` as on branch-4.x: a
    deduplicate primary-key table in Parquet, written in two snapshots. The
    second updates ids 1 and 2 and inserts id 4; id 3 holds a root array.
    - The JNI section of `test_paimon_catalog_variant` gets the three
    branch-4.x blocks: the whole Variant value, Variant paths cast to
    scalars, and a predicate on a Variant path. All three expect the latest
    value of ids 1 and 2.
    
    The other parts of #66466 and #68242 do not apply to master:
    
    | Branch-4.x change | Master |
    |---|---|
    | FE rejects a Variant projection while Variant V2 is disabled | #66858
    made Variant V2 the default and removed the switch, so there is nothing
    to check |
    | BE rejects a physical Variant read from any format but Parquet | Would
    reject the native Variant reads of the ADBC reader (#68883). Master
    keeps the Iceberg-specific gate in `IcebergTableReader`, admits Arrow
    Variant reads in `FileScannerV2` only for ADBC, and plans a Paimon split
    that projects a Variant natively only when every file is Parquet or an
    ORC file whose schema predates the Variant column. Paimon 1.4.2 cannot
    create an ORC table with a VARIANT column (`Unsupported type: VARIANT`)
    |
    | WAL is exempted from that gate | WAL is not gated on master |
    
    **Results.** On a local cluster against the Iceberg docker environment,
    Spark reads `variant_primary_key_smoke` as four rows, with the
    second-snapshot values for ids 1 and 2. Doris returns the same rows in
    the three new blocks, and every other result block of the suite is
    unchanged.
    
    Co-authored-by: zhangstar333 <zh...@selectdb.com>
---
 .../create_preinstalled_scripts/paimon/run13.sql   | 26 ++++++++++++++++++++++
 .../paimon/test_paimon_catalog_variant.out         | 16 +++++++++++++
 .../paimon/test_paimon_catalog_variant.groovy      | 26 +++++++++++++++++++++-
 3 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
index ec071966382..49ff2d2369d 100644
--- a/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
+++ b/docker/thirdparties/docker-compose/iceberg/scripts/create_preinstalled_scripts/paimon/run13.sql
@@ -70,3 +70,29 @@ alter table variant_mixed_su unset tblproperties (
 );
 insert into variant_mixed_su values
     (2, date '2026-07-01', parse_json('{"name":"bob","age":30,"layout":"unshredded"}'));
+
+-- Keep a primary-key table separate from variant_smoke so the regression covers both Paimon
+-- append-only reads and deduplicate merge reads with Variant payloads.
+drop table if exists variant_primary_key_smoke;
+create table variant_primary_key_smoke (
+    id BIGINT,
+    payload VARIANT
+) using paimon
+tblproperties (
+    'primary-key' = 'id',
+    'bucket' = '1',
+    'merge-engine' = 'deduplicate',
+    'file.format' = 'parquet'
+);
+
+insert into variant_primary_key_smoke values
+    (1, parse_json('{"name":"alice","version":1,"active":false,"profile":{"city":"beijing"}}')),
+    (2, parse_json('{"name":"bob","version":1,"tags":["old"]}')),
+    (3, parse_json('[10,"original"]'));
+
+-- A second snapshot updates two existing keys and inserts a new key. Doris should expose only the
+-- latest logical row for ids 1 and 2.
+insert into variant_primary_key_smoke values
+    (1, parse_json('{"name":"alice-updated","version":2,"active":true,"profile":{"city":"hangzhou"}}')),
+    (2, parse_json('{"name":"bob","version":2,"tags":["new","primary-key"]}')),
+    (4, parse_json('{"name":"carol","version":1,"active":true}'));
diff --git a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
index 256a115c8d2..d1e70d1b963 100644
--- a/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
+++ b/regression-test/data/external_table_p0/paimon/test_paimon_catalog_variant.out
@@ -24,6 +24,22 @@ payload	variant<PROPERTIES ("variant_max_subcolumns_count" = "0","variant_enable
 -- !subpath_predicate --
 2	bob
 
+-- !primary_key_full_variant --
+1	{"active":true,"name":"alice-updated","profile":{"city":"hangzhou"},"version":2}
+2	{"name":"bob","tags":["new","primary-key"],"version":2}
+3	[10,"original"]
+4	{"active":true,"name":"carol","version":1}
+
+-- !primary_key_deduplicate --
+1	alice-updated	2	true
+2	bob	2	\N
+3	\N	\N	\N
+4	carol	1	true
+
+-- !primary_key_subpath_predicate --
+1	alice-updated
+2	bob
+
 -- !jni_shredded_projection --
 2	bob	30
 
diff --git a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
index f8364971e8f..8aa1933acc7 100644
--- a/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
+++ b/regression-test/suites/external_table_p0/paimon/test_paimon_catalog_variant.groovy
@@ -106,8 +106,32 @@ suite("test_paimon_catalog_variant", "p0,external,doris,external_docker,external
             order by id
         """
 
+        // variant_smoke is append-only. This table has duplicate primary-key writes, so the scan
+        // must preserve Paimon's deduplicate semantics while materializing Variant values.
+        order_qt_primary_key_full_variant """
+            select id, payload
+            from variant_primary_key_smoke
+            order by id
+        """
+
+        order_qt_primary_key_deduplicate """
+            select id,
+                   cast(payload['name'] as string),
+                   cast(payload['version'] as int),
+                   cast(payload['active'] as boolean)
+            from variant_primary_key_smoke
+            order by id
+        """
+
+        order_qt_primary_key_subpath_predicate """
+            select id, cast(payload['name'] as string)
+            from variant_primary_key_smoke
+            where cast(payload['version'] as int) = 2
+            order by id
+        """
+
         // Exercise Paimon's metadata-marked read type against a physically shredded file. The
-        // variant_smoke cases above read unshredded files.
+        // unshredded and primary-key cases above cover Paimon's raw-value and merge fallbacks.
         order_qt_jni_shredded_projection """
             select id,
                    cast(payload['name'] as string),


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org