You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by cs...@apache.org on 2026/07/01 12:53:14 UTC

(impala) branch master updated (3057fda80 -> e47134ece)

This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


    from 3057fda80 IMPALA-14484: Calcite planner: Add column stats to union slot descriptors
     new b4a4c6566 IMPALA-14705: [DOCS] Document Iceberg DML
     new e47134ece IMPALA-15126: Fix NullPointerException for STRUCT.* expansion on masked tables

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 docs/impala_keydefs.ditamap                        |  1 +
 docs/topics/impala_dml.xml                         | 14 ++++++---
 docs/topics/impala_merge.xml                       |  2 +-
 .../main/java/org/apache/impala/analysis/Path.java | 12 ++++++++
 .../authorization/AuthorizationStmtTest.java       | 32 +++++++++++++++++++
 .../ranger_column_masking_complex_types.test       | 36 ++++++++++++++++++++++
 6 files changed, 92 insertions(+), 5 deletions(-)


(impala) 02/02: IMPALA-15126: Fix NullPointerException for STRUCT.* expansion on masked tables

Posted by cs...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit e47134ece299ddd2d538c9017b44b383b35e9d28
Author: Zoltan Borok-Nagy <bo...@cloudera.com>
AuthorDate: Tue Jun 30 15:22:43 2026 +0200

    IMPALA-15126: Fix NullPointerException for STRUCT.* expansion on masked tables
    
    When a table has a column-masking policy, queries are rewritten to read
    from a table-masking view and re-analyzed. Expanding a struct that
    contains a nested collection - via STRUCT.* or SELECT * with
    EXPAND_COMPLEX_TYPES=true - built the collection's relative path with
    Path.createRelPath(), which did not carry the path's table-masking
    marker. The collection was therefore rooted at the underlying table's
    tuple, which is registered only inside the masking view's analyzer, so
    CollectionTableRef.analyze() -> Analyzer.findAnalyzer() returned null
    and threw a NullPointerException.
    
    Path.createRelPath() now propagates the pre-masking path to the derived
    relative path, so a collection reached through struct expansion roots at
    the masking view's tuple in the current query block (matching how
    directly selected masked collections already resolve).
    
    Testing:
    - New FE test
      AuthorizationStmtTest.testColumnMaskingStructWithNestedCollection
    - New e2e test cases in
      QueryTest/ranger_column_masking_complex_types.test
    
    Change-Id: Id101542aed900220383481f17d812b1889f78736
    Assisted-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
    Reviewed-on: http://gerrit.cloudera.org:8080/24524
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 .../main/java/org/apache/impala/analysis/Path.java | 12 ++++++++
 .../authorization/AuthorizationStmtTest.java       | 32 +++++++++++++++++++
 .../ranger_column_masking_complex_types.test       | 36 ++++++++++++++++++++++
 3 files changed, 80 insertions(+)

diff --git a/fe/src/main/java/org/apache/impala/analysis/Path.java b/fe/src/main/java/org/apache/impala/analysis/Path.java
index d1623013c..040a2869e 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Path.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Path.java
@@ -652,6 +652,18 @@ public class Path {
   public static Path createRelPath(Path rootPath, String... fieldNames) {
     Preconditions.checkState(rootPath.isResolved());
     Path result = new Path(rootPath, Arrays.asList(fieldNames));
+    // Propagate the table-masking marker (column masking). Without this, a collection
+    // reached through this relative path - e.g. via '*'/STRUCT.* expansion, possibly
+    // nested in structs - would root at the underlying table's tuple, which is only
+    // registered inside the table-masking view's analyzer. CollectionTableRef.analyze()
+    // -> Analyzer.findAnalyzer() would then return null and hit a NullPointerException
+    // (IMPALA-15126). Rooting at the pre-masking path keeps it in the current block.
+    if (rootPath.isMaskedPath()) {
+      Path relPathBeforeMasking =
+          new Path(rootPath.getPathBeforeMasking(), Arrays.asList(fieldNames));
+      Preconditions.checkState(relPathBeforeMasking.resolve());
+      result.setPathBeforeMasking(relPathBeforeMasking);
+    }
     return result;
   }
 }
diff --git a/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java b/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
index 2699dc4b2..5eeb07c06 100644
--- a/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
+++ b/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
@@ -3337,6 +3337,38 @@ public class AuthorizationStmtTest extends AuthorizationTestBase {
     }
   }
 
+  /**
+   * IMPALA-15126: Expanding a struct that contains a nested collection (via STRUCT.*
+   * or SELECT * with EXPAND_COMPLEX_TYPES=true) on a table that has a column-masking
+   * policy used to throw a NullPointerException during analysis. The collection nested
+   * in the struct was rooted at the underlying table's tuple, which is only registered
+   * inside the table-masking view, so Analyzer.findAnalyzer() returned null.
+   */
+  @Test
+  public void testColumnMaskingStructWithNestedCollection() throws ImpalaException {
+    String policyName = "complextypestbl_id_mask";
+    // EXPAND_COMPLEX_TYPES makes '*' / 'struct.*' include complex-typed fields.
+    AnalysisContext ctx = createAnalysisCtx(authzFactory_, user_.getName());
+    ctx.getQueryOptions().setExpand_complex_types(true);
+    try {
+      // A mask on a scalar column wraps the table in a table-masking view.
+      createColumnMaskingPolicy(policyName, "functional_parquet", "complextypestbl",
+          "id", user_.getShortName(), "CUSTOM", "100 * {col}");
+      rangerImpalaPlugin_.refreshPoliciesAndTags();
+
+      // 'nested_struct' contains nested collections; expanding it via STRUCT.* used to
+      // hit a NullPointerException under masking.
+      authorize(ctx,
+          "select id, nested_struct.* from functional_parquet.complextypestbl")
+          .ok(onServer(TPrivilegeLevel.ALL));
+      // The same struct is expanded by SELECT * when EXPAND_COMPLEX_TYPES=true.
+      authorize(ctx, "select * from functional_parquet.complextypestbl")
+          .ok(onServer(TPrivilegeLevel.ALL));
+    } finally {
+      deleteRangerPolicy(policyName);
+    }
+  }
+
   private void createRowFilteringPolicy(String policyName, String dbName, String tblName,
       String user, String rowFilter) {
     String json = String.format("{\n" +
diff --git a/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test b/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test
index efa1cce88..fe6ab18d4 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test
@@ -91,6 +91,42 @@ NULL,'NULL','NULL','NULL'
 INT,STRING,STRING,STRING
 ====
 ---- QUERY
+# IMPALA-15126: Expanding a struct that contains a nested collection via STRUCT.* on a
+# column-masked table used to throw a NullPointerException. EXPAND_COMPLEX_TYPES is set
+# explicitly so the regression is covered regardless of the option's default.
+set EXPAND_COMPLEX_TYPES=1;
+select id, nested_struct.* from complextypestbl
+---- RESULTS
+800,-1,'[-1]','{"d":[[{"e":-1,"f":"nonnullable"}]]}','{}'
+100,1,'[1]','{"d":[[{"e":10,"f":"aaa"},{"e":-10,"f":"bbb"}],[{"e":11,"f":"c"}]]}','{"foo":{"h":{"i":[1.1]}}}'
+200,NULL,'[null]','{"d":[[{"e":null,"f":null},{"e":10,"f":"aaa"},{"e":null,"f":null},{"e":-10,"f":"bbb"},{"e":null,"f":null}],[{"e":11,"f":"c"},null],[],null]}','{"g1":{"h":{"i":[2.2,null]}},"g2":{"h":{"i":[]}},"g3":null,"g4":{"h":{"i":null}},"g5":{"h":null}}'
+300,NULL,'NULL','{"d":[]}','{}'
+400,NULL,'NULL','{"d":null}','NULL'
+500,NULL,'NULL','NULL','{"foo":{"h":{"i":[2.2,3.3]}}}'
+600,NULL,'NULL','NULL','NULL'
+700,7,'[2,3,null]','{"d":[[],[null],null]}','NULL'
+---- TYPES
+BIGINT, INT, STRING, STRING, STRING
+====
+---- QUERY
+# IMPALA-15126: SELECT * on a column-masked table whose struct columns contain nested
+# collections. The scalar 'id' mask still applies (100*id); masks on complex columns are
+# no-ops, so the complex columns are returned unmasked.
+set EXPAND_COMPLEX_TYPES=1;
+select * from complextypestbl
+---- RESULTS
+100,'[1,2,3]','[[1,2],[3,4]]','{"k1":1,"k2":100}','[{"k1":1}]','{"a":1,"b":[1],"c":{"d":[[{"e":10,"f":"aaa"},{"e":-10,"f":"bbb"}],[{"e":11,"f":"c"}]]},"g":{"foo":{"h":{"i":[1.1]}}}}'
+200,'[null,1,2,null,3,null]','[[null,1,2,null],[3,null,4],[],null]','{"k1":2,"k2":null}','[{"k3":null,"k1":1},null,{}]','{"a":null,"b":[null],"c":{"d":[[{"e":null,"f":null},{"e":10,"f":"aaa"},{"e":null,"f":null},{"e":-10,"f":"bbb"},{"e":null,"f":null}],[{"e":11,"f":"c"},null],[],null]},"g":{"g1":{"h":{"i":[2.2,null]}},"g2":{"h":{"i":[]}},"g3":null,"g4":{"h":{"i":null}},"g5":{"h":null}}}'
+300,'[]','[null]','{}','[null,null]','{"a":null,"b":null,"c":{"d":[]},"g":{}}'
+400,'NULL','[]','{}','[]','{"a":null,"b":null,"c":{"d":null},"g":null}'
+500,'NULL','NULL','{}','NULL','{"a":null,"b":null,"c":null,"g":{"foo":{"h":{"i":[2.2,3.3]}}}}'
+600,'NULL','NULL','NULL','NULL','NULL'
+700,'NULL','[null,[5,6]]','{"k1":null,"k3":null}','NULL','{"a":7,"b":[2,3,null],"c":{"d":[[],[null],null]},"g":null}'
+800,'[-1]','[[-1,-2],[]]','{"k1":-1}','[{},{"k1":1},{},{}]','{"a":-1,"b":[-1],"c":{"d":[[{"e":-1,"f":"nonnullable"}]]},"g":{}}'
+---- TYPES
+BIGINT, STRING, STRING, STRING, STRING, STRING
+====
+---- QUERY
 # Test resolving explicit STAR path on a nested struct column inside array
 select id, nested_arr.item.*
 from complextypestbl t,


(impala) 01/02: IMPALA-14705: [DOCS] Document Iceberg DML

Posted by cs...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit b4a4c656615485dee337fc3e5f4a58c13a78b260
Author: Gowthami Bisati <gb...@gmail.com>
AuthorDate: Wed Jun 24 15:03:57 2026 +0530

    IMPALA-14705: [DOCS] Document Iceberg DML
    
    Document DELETE, UPDATE and MERGE statements support for Iceberg tables.
    
    What changed:
    
    Updated docs/topics/impala_dml.xml to document DELETE, UPDATE AND MERGE
    statements for Iceberg tables.
    
    Updated docs/impala_keydefs.ditamap to add the key to merge statement.
    
    Updated the docs/topics/impala_merge.xml to add the title for Merge
    statement with the Impala version details.
    
    Tests performed:
    
    make all
    ./pre-commit.sh
    xmllint --noout docs/topics/impala_dml.xml
    git diff docs/topics/impala_dml.xml
    
    All commands completed successfully. make all generated the XHTML, HTML5,
    and PDF documentation outputs.
    
    Change-Id: Idda990274962dee57e345efb59147b2455938c3f
    Reviewed-on: http://gerrit.cloudera.org:8080/24503
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 docs/impala_keydefs.ditamap  |  1 +
 docs/topics/impala_dml.xml   | 14 ++++++++++----
 docs/topics/impala_merge.xml |  2 +-
 3 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/docs/impala_keydefs.ditamap b/docs/impala_keydefs.ditamap
index b0f3b51f9..125909724 100644
--- a/docs/impala_keydefs.ditamap
+++ b/docs/impala_keydefs.ditamap
@@ -10876,6 +10876,7 @@ under the License.
   <keydef href="topics/impala_truncate_table.xml" keys="truncate_table"/>
   <keydef href="topics/impala_update.xml" keys="update"/>
   <keydef href="topics/impala_upsert.xml" keys="upsert"/>
+  <keydef href="topics/impala_merge.xml" keys="merge"/>
   <keydef href="topics/impala_use.xml" keys="use"/>
   <keydef href="topics/impala_functions.xml" keys="builtins"/>
   <keydef href="topics/impala_math_functions.xml" keys="math_functions"/>
diff --git a/docs/topics/impala_dml.xml b/docs/topics/impala_dml.xml
index 02cc002bc..419d153a7 100644
--- a/docs/topics/impala_dml.xml
+++ b/docs/topics/impala_dml.xml
@@ -44,7 +44,7 @@ under the License.
 
     <ul>
       <li>
-        <xref keyref="delete"/>. Works for Kudu tables only.
+        <xref keyref="delete"/>. Works for Kudu and Iceberg tables.
       </li>
 
       <li>
@@ -56,12 +56,16 @@ under the License.
       </li>
 
       <li>
-        <xref keyref="update"/>. Works for Kudu tables only.
+        <xref keyref="update"/>. Works for Kudu and Iceberg tables.
       </li>
 
       <li>
         <xref keyref="upsert"/>. Works for Kudu tables only.
       </li>
+
+      <li>
+        <xref keyref="merge"/>. Works for Iceberg tables only.
+      </li>
     </ul>
 
     <p>
@@ -80,8 +84,10 @@ under the License.
     </p>
 
     <p>
-      In <keyword keyref="impala28_full"/> and higher, Impala does support the <codeph>UPDATE</codeph>, <codeph>DELETE</codeph>,
-      and <codeph>UPSERT</codeph> statements for Kudu tables.
+      In <keyword keyref="impala28_full"/> and higher, Impala supports the <codeph>UPDATE</codeph>, <codeph>DELETE</codeph>, 
+      and <codeph>UPSERT</codeph> statements for Kudu tables. For Iceberg V2 tables, Impala supports the <codeph>DELETE</codeph> 
+      statement starting in <keyword keyref="impala43"/>, the <codeph>UPDATE</codeph> statement starting in <keyword keyref="impala44"/>, 
+      and the <codeph>MERGE</codeph> statement starting in <keyword keyref="impala45"/>.
       For HDFS or S3 tables, to simulate the effects of an <codeph>UPDATE</codeph> or <codeph>DELETE</codeph> statement
       in other database systems, typically you use <codeph>INSERT</codeph> or <codeph>CREATE TABLE AS SELECT</codeph> to copy data
       from one table to another, filtering out or changing the appropriate rows during the copy operation.
diff --git a/docs/topics/impala_merge.xml b/docs/topics/impala_merge.xml
index 1afd47147..fa135e94c 100644
--- a/docs/topics/impala_merge.xml
+++ b/docs/topics/impala_merge.xml
@@ -20,7 +20,7 @@ under the License.
 <!DOCTYPE concept PUBLIC "-//OASIS//DTD DITA Concept//EN" "concept.dtd">
 <concept id="merge">
 
-  <title>MERGE Statement</title>
+  <title>MERGE Statement (<keyword keyref="impala45"/> or higher only)</title>
   <titlealts audience="PDF"><navtitle>MERGE</navtitle></titlealts>
   <prolog>
     <metadata>