You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Daniel Becker (Code Review)" <ge...@cloudera.org> on 2025/07/01 09:12:21 UTC

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23107


Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................

IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

When trying to doubly unnest a 2D array from an Iceberg table that has
delete files but not for every data file, we run into an error:

  Filtering an unnested collection that comes from a UNION [ALL] is not
  supported yet.

This is because there is a UNION node because of the Iceberg delete
files, and there is an added "not-empty" conjunct on the collections.

IMPALA-12753 describes a bug where a conjunct on an unnested collection
coming from a UNION ALL is only applied to the first UNION operand. To
avoid incorrectness, we disabled this case in the commit for
IMPALA-12695, but its unintended consequence is that it leads to this
error with Iceberg tables.

However, in this case with Iceberg deletes, the bug described in
IMPALA-12753 is not present because both sides of the UNION have the
same tuple id, so conjuncts are naturally applied to both sides.

This commit relaxes the check, which now does not fire if all UNION
operands have the same tuple ids.

Testing:
 - existing tests related to IMPALA-12753 pass
 - added a regression test with an Iceberg table with DELETE files

Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
---
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M tests/query_test/test_nested_types.py
2 files changed, 34 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/23107/2
-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12100/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 13:32:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 2:

(1 comment)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/23107/2/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:

http://gerrit.cloudera.org:8080/#/c/23107/2/tests/query_test/test_nested_types.py@215
PS2, Line 215: ;
flake8: E703 statement ends with a semicolon



-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 09:14:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/23107/2/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:

http://gerrit.cloudera.org:8080/#/c/23107/2/tests/query_test/test_nested_types.py@196
PS2, Line 196: 
should have  @SkipIfFS.hive


http://gerrit.cloudera.org:8080/#/c/23107/2/tests/query_test/test_nested_types.py@214
PS2, Line 214:     self.execute_query_expect_success(self.client,
it would be nice to check the results too



-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 12:42:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19655/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 13:28:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................

IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

When trying to doubly unnest a 2D array from an Iceberg table that has
delete files but not for every data file, we run into an error:

  Filtering an unnested collection that comes from a UNION [ALL] is not
  supported yet.

This is because there is a UNION node because of the Iceberg delete
files, and there is an added "not-empty" conjunct on the collections.

IMPALA-12753 describes a bug where a conjunct on an unnested collection
coming from a UNION ALL is only applied to the first UNION operand. To
avoid incorrectness, we disabled this case in the commit for
IMPALA-12695, but its unintended consequence is that it leads to this
error with Iceberg tables.

However, in this case with Iceberg deletes, the bug described in
IMPALA-12753 is not present because both sides of the UNION have the
same tuple id, so conjuncts are naturally applied to both sides.

This commit relaxes the check, which now does not fire if all UNION
operands have the same tuple ids.

Testing:
 - existing tests related to IMPALA-12753 pass
 - added a regression test with an Iceberg table with DELETE files

Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
---
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M tests/query_test/test_nested_types.py
2 files changed, 37 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/23107/3
-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 13:29:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................

IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

When trying to doubly unnest a 2D array from an Iceberg table that has
delete files but not for every data file, we run into an error:

  Filtering an unnested collection that comes from a UNION [ALL] is not
  supported yet.

This is because there is a UNION node because of the Iceberg delete
files, and there is an added "not-empty" conjunct on the collections.

IMPALA-12753 describes a bug where a conjunct on an unnested collection
coming from a UNION ALL is only applied to the first UNION operand. To
avoid incorrectness, we disabled this case in the commit for
IMPALA-12695, but its unintended consequence is that it leads to this
error with Iceberg tables.

However, in this case with Iceberg deletes, the bug described in
IMPALA-12753 is not present because both sides of the UNION have the
same tuple id, so conjuncts are naturally applied to both sides.

This commit relaxes the check, which now does not fire if all UNION
operands have the same tuple ids.

Testing:
 - existing tests related to IMPALA-12753 pass
 - added a regression test with an Iceberg table with DELETE files

Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Reviewed-on: http://gerrit.cloudera.org:8080/23107
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/planner/UnnestNode.java
M tests/query_test/test_nested_types.py
2 files changed, 37 insertions(+), 0 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 13:32:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19653/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 09:38:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23107 )

Change subject: IMPALA-14185: Error unnesting nested array from Iceberg with DELETE files
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbc6f580586d4b337f33a2f32052aa07f6fca828
Gerrit-Change-Number: 23107
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jul 2025 19:49:53 +0000
Gerrit-HasComments: No