You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org> on 2026/09/18 09:12:55 UTC

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24881


Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................

IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

'v IS NULL' and 'v IS NOT NULL' were rejected during analysis for
VARIANT columns. The reason was that there was no is_null_pred()
builtin for VARIANT: the builtins are registered for the types of
Type.getSupportedTypes(), which are all scalar, and when VARIANT read
support was added a VARIANT had no expression ABI yet. So the check
only turned a "no matching function" error into a clearer message.

Since IMPALA-15057 a VARIANT is a self-contained value (VariantVal) at
the UDF boundary, just like the scalar types, therefore the builtins
can simply be instantiated for it.

Only SQL NULL is NULL. A variant null in the column is a regular value,
so 'v IS NULL' is false for it. Note that variant_get() does return SQL
NULL when there is a variant null at the path, hence
variant_get(v, '$') IS NULL can differ from v IS NULL.

The predicate is pushed down to Iceberg the same way as for the other
types. This is safe: Iceberg only uses it to prune whole files by their
NULL counts, which it derives from the required metadata column of the
VARIANT, and it never replaces the evaluation of the predicate.

Testing:
* FE: extended AnalyzeExprsTest.TestIsNullPredicates
* e2e tests added

Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Assisted-by: Claude Fable 5.1 (Claude Code)
---
M be/src/exprs/is-null-predicate-ir.cc
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test
6 files changed, 105 insertions(+), 16 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/24881/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................

IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

'v IS NULL' and 'v IS NOT NULL' were rejected during analysis for
VARIANT columns. The reason was that there was no is_null_pred()
builtin for VARIANT: the builtins are registered for the types of
Type.getSupportedTypes(), which are all scalar, and when VARIANT read
support was added a VARIANT had no expression ABI yet. So the check
only turned a "no matching function" error into a clearer message.

Since IMPALA-15057 a VARIANT is a self-contained value (VariantVal) at
the UDF boundary, just like the scalar types, therefore the builtins
can simply be instantiated for it.

Only SQL NULL is NULL. A variant null in the column is a regular value,
so 'v IS NULL' is false for it. Note that variant_get() does return SQL
NULL when there is a variant null at the path, hence
variant_get(v, '$') IS NULL can differ from v IS NULL.

The predicate is pushed down to Iceberg the same way as for the other
types. This is safe: Iceberg only uses it to prune whole files by their
NULL counts, which it derives from the required metadata column of the
VARIANT, and it never replaces the evaluation of the predicate.

Testing:
* FE: extended AnalyzeExprsTest.TestIsNullPredicates
* e2e tests added

Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Assisted-by: Claude Fable 5.1 (Claude Code)
---
M be/src/exprs/is-null-predicate-ir.cc
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test
7 files changed, 112 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/24881/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14080/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 14:08:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24081/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 09:39:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/24351/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 15:55:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 5
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 06 Oct 2026 17:15:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 2: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24881/2/fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
File fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java:

http://gerrit.cloudera.org:8080/#/c/24881/2/fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java@66
PS2, Line 66:     List<Type> types = new ArrayList<>(Type.getSupportedTypes());
Isn't it better to add it ad Type.java?



-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 12:29:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 2:

(1 comment)

Thanks for the review!

http://gerrit.cloudera.org:8080/#/c/24881/2/fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
File fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java:

http://gerrit.cloudera.org:8080/#/c/24881/2/fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java@66
PS2, Line 66:     List<Type> types = new ArrayList<>(Type.getSupportedTypes());
> Isn't it better to add it ad Type.java?
Supported types only contain scalar types, and curently VARIANT is not a scalar type. Allowing non-scalar types in supported types could make sense, but it needs a somewhat bigger change.

Also not that future-proof as there could be other cases where we loop over supported types and accidentally include VARIANT when we should exclude it.

But I can give it a try if you feel strong about it.



-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2026 17:12:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14077/


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 13:44:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................

IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

'v IS NULL' and 'v IS NOT NULL' were rejected during analysis for
VARIANT columns. The reason was that there was no is_null_pred()
builtin for VARIANT: the builtins are registered for the types of
Type.getSupportedTypes(), which are all scalar, and when VARIANT read
support was added a VARIANT had no expression ABI yet. So the check
only turned a "no matching function" error into a clearer message.

Since IMPALA-15057 a VARIANT is a self-contained value (VariantVal) at
the UDF boundary, just like the scalar types, therefore the builtins
can simply be instantiated for it.

Only SQL NULL is NULL. A variant null in the column is a regular value,
so 'v IS NULL' is false for it. Note that variant_get() does return SQL
NULL when there is a variant null at the path, hence
variant_get(v, '$') IS NULL can differ from v IS NULL.

The predicate is pushed down to Iceberg the same way as for the other
types. This is safe: Iceberg only uses it to prune whole files by their
NULL counts, which it derives from the required metadata column of the
VARIANT, and it never replaces the evaluation of the predicate.

Testing:
* FE: extended AnalyzeExprsTest.TestIsNullPredicates
* e2e tests added

Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Assisted-by: Claude Fable 5.1 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24881
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exprs/is-null-predicate-ir.cc
M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java
M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-v3-variant.test
7 files changed, 112 insertions(+), 16 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 6
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14077/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 09:13:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-15371: Support IS [NOT] NULL on VARIANT columns

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24881 )

Change subject: IMPALA-15371: Support IS [NOT] NULL on VARIANT columns
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14080/


-- 
To view, visit http://gerrit.cloudera.org:8080/24881
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id0b17821f6c8a988abf93494b7dbeb8f6f7db216
Gerrit-Change-Number: 24881
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Fri, 18 Sep 2026 18:32:34 +0000
Gerrit-HasComments: No