You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org> on 2022/05/02 08:27:43 UTC

[Impala-ASF-CR] IMPALA-9410: WIP

Gergely Fürnstáhl has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18397


Change subject: IMPALA-9410: WIP
......................................................................

IMPALA-9410: WIP

Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M tests/query_test/test_nested_types.py
8 files changed, 319 insertions(+), 236 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/18397/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 1
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.

Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 6:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.h
File be/src/exec/orc-metadata-utils.h:

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.h@109
PS6, Line 109: std::string
> This is still unresolved.
Done


http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.cc
File be/src/exec/orc-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.cc@247
PS6, Line 247: std::string
> could be const&
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jun 2022 13:37:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10646/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 10
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 May 2022 13:45:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8179/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 13
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Jun 2022 12:31:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................

IMPALA-9410: Support resolving ORC file columns by names

Added query option and implementation to be able to resolve columns by
names.

Changed secondary resolution strategy for iceberg orc tables to name
based resolution.

Testing:

Added new test dimension for orc tests, added results to now working
iceberg migrated table test

Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Reviewed-on: http://gerrit.cloudera.org:8080/18397
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
M tests/common/test_dimensions.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_scanners.py
11 files changed, 378 insertions(+), 245 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 14
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.

Gergely Fürnstáhl has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 6:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG@12
PS5, Line 12: strategy
> typo: strategy
Done


http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG@14
PS5, Line 14: 
> Could you addd "Testing:" before the paragraph that is about testing? This 
Done


http://gerrit.cloudera.org:8080/#/c/18397/5/be/src/exec/orc-metadata-utils.cc
File be/src/exec/orc-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/18397/5/be/src/exec/orc-metadata-utils.cc@47
PS5, Line 47: DetermineFullAcidSchema();
> nit: it could be added to the init-list.
Done


http://gerrit.cloudera.org:8080/#/c/18397/5/be/src/exec/orc-metadata-utils.cc@184
PS5, Line 184: 
> at this level table_idx can be -1.
Done


http://gerrit.cloudera.org:8080/#/c/18397/5/testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test:

http://gerrit.cloudera.org:8080/#/c/18397/5/testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test@77
PS5, Line 77: lect * from functional_parquet.iceberg_
> nit: This won't fail anymore.
Done


http://gerrit.cloudera.org:8080/#/c/18397/3/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:

http://gerrit.cloudera.org:8080/#/c/18397/3/tests/query_test/test_nested_types.py@138
PS3, Line 138:     cls.ImpalaTestMatrix.add_dimension(
> We should add a constraint (see e.g. at L136) to not run the tests with bot
Done


http://gerrit.cloudera.org:8080/#/c/18397/5/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:

http://gerrit.cloudera.org:8080/#/c/18397/5/tests/query_test/test_nested_types.py@64
PS5, Line 64:     cls.ImpalaTestMatrix.add_dimension(ImpalaTestDimension('orc_schema_resolution', 0, 1))
> We could also add this dimension to some tests in test_scanners.py as well.
Done


http://gerrit.cloudera.org:8080/#/c/18397/5/tests/query_test/test_nested_types.py@65
PS5, Line 65: on
> missing argument, the above method expects 'vector'
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 May 2022 10:02:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.h
File be/src/exec/orc-metadata-utils.h:

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.h@109
PS6, Line 109: std::string
> could be const&
This is still unresolved.



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jun 2022 08:38:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.

Gergely Fürnstáhl has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................

IMPALA-9410: Support resolving ORC file columns by names

Added query option and implementation to be able to resolve columns by
names.

Changed secondary resolution startegy for iceberg orc tables to name
based resolution.

Added new test dimension for orc tests, added results to now working
iceberg migrated table test

Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
M tests/query_test/test_nested_types.py
9 files changed, 350 insertions(+), 243 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/18397/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 13: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 13
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Jun 2022 17:06:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 13: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 13
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 02 Jun 2022 12:31:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 6: Code-Review+1

(2 comments)

Looks great!

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.h
File be/src/exec/orc-metadata-utils.h:

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.h@109
PS6, Line 109: std::string
could be const&


http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.cc
File be/src/exec/orc-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/18397/6/be/src/exec/orc-metadata-utils.cc@247
PS6, Line 247: std::string
could be const&



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 May 2022 10:19:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10546/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 May 2022 15:39:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Tamas Mate (Code Review)" <ge...@cloudera.org>.

Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 5: Code-Review+1

(2 comments)

LGTM!

http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG@14
PS5, Line 14: 
Could you addd "Testing:" before the paragraph that is about testing? This is just a small commit style nit.


http://gerrit.cloudera.org:8080/#/c/18397/5/testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
File testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test:

http://gerrit.cloudera.org:8080/#/c/18397/5/testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test@77
PS5, Line 77: Currently this fails due to IMPALA-9410
nit: This won't fail anymore.



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 May 2022 08:59:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10615/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 6
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 May 2022 10:22:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.

Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 5:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18397/5//COMMIT_MSG@12
PS5, Line 12: startegy
typo: strategy


http://gerrit.cloudera.org:8080/#/c/18397/5/be/src/exec/orc-metadata-utils.cc
File be/src/exec/orc-metadata-utils.cc:

http://gerrit.cloudera.org:8080/#/c/18397/5/be/src/exec/orc-metadata-utils.cc@47
PS5, Line 47: schema_resolution_strategy_ = schema_resolution
nit: it could be added to the init-list.


http://gerrit.cloudera.org:8080/#/c/18397/5/be/src/exec/orc-metadata-utils.cc@184
PS5, Line 184: table_idx
at this level table_idx can be -1.

E.g. the following crashes Impala:

 set orc_schema_resolution=name;
 select id from alltypestiny;


http://gerrit.cloudera.org:8080/#/c/18397/5/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:

http://gerrit.cloudera.org:8080/#/c/18397/5/tests/query_test/test_nested_types.py@64
PS5, Line 64:     cls.ImpalaTestMatrix.add_dimension(ImpalaTestDimension('orc_schema_resolution', 0, 1))
We could also add this dimension to some tests in test_scanners.py as well.


http://gerrit.cloudera.org:8080/#/c/18397/5/tests/query_test/test_nested_types.py@65
PS5, Line 65: ()
missing argument, the above method expects 'vector'



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 5
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 May 2022 13:35:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/10672/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jun 2022 14:30:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.

Gergely Fürnstáhl has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................

IMPALA-9410: Support resolving ORC file columns by names

Added query option and implementation to be able to resolve columns by
names.

Changed secondary resolution strategy for iceberg orc tables to name
based resolution.

Testing:

Added new test dimension for orc tests, added results to now working
iceberg migrated table test

Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
M tests/common/test_dimensions.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_scanners.py
11 files changed, 377 insertions(+), 245 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/18397/11
-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 11
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Gergely Fürnstáhl (Code Review)" <ge...@cloudera.org>.

Gergely Fürnstáhl has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................

IMPALA-9410: Support resolving ORC file columns by names

Added query option and implementation to be able to resolve columns by
names.

Changed secondary resolution strategy for iceberg orc tables to name
based resolution.

Testing:

Added new test dimension for orc tests, added results to now working
iceberg migrated table test

Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M testdata/workloads/functional-query/queries/QueryTest/iceberg-migrated-tables.test
M tests/common/test_dimensions.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_scanners.py
11 files changed, 378 insertions(+), 245 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/18397/12
-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 12
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-9410: Support resolving ORC file columns by names

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18397 )

Change subject: IMPALA-9410: Support resolving ORC file columns by names
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18397/7/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:

http://gerrit.cloudera.org:8080/#/c/18397/7/tests/query_test/test_nested_types.py@39
PS7, Line 39: e
flake8: E501 line too long (111 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/18397
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29562a059160c19eb58ccea76aa959d2e408f8de
Gerrit-Change-Number: 18397
Gerrit-PatchSet: 7
Gerrit-Owner: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Gergely Fürnstáhl <gf...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tamas Mate <tm...@apache.org>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 May 2022 15:17:11 +0000
Gerrit-HasComments: Yes