You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2019/07/17 13:32:15 UTC
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13874
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES on a cold started cluster. With this patch,
the time reduces to ~1s.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
10 files changed, 106 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/4
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 22 Jul 2019 01:16:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 12: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 01:03:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4652/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 01:03:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 12: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4652/
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 07:58:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 9: Code-Review+2
Thanks for fixing this.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 9
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 6:
(2 comments)
Have some trouble setting up HUE in my local machine. Address some comments first.
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java
File fe/src/main/java/org/apache/impala/catalog/Catalog.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java@167
PS5, Line 167: Catalog v1
> Right, I'm talking about the terminology. In the code, we never see these t
Sure. Please correct the updated comment if it's still inaccurate :)
http://gerrit.cloudera.org:8080/#/c/13874/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
File fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/13874/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@385
PS6, Line 385: tbl == null ||
> Why this
Sorry, don't need this
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 22 Jul 2019 00:33:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES for all tables on a cold started cluster. With
this patch, the time reduces to ~1s.
- Testing in HUE-4.4.0 with a db with 3000 tables, the performance is the
same as using legacy catalog implementation.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Reviewed-on: http://gerrit.cloudera.org:8080/13874
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 156 insertions(+), 15 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 14
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4637/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 12:41:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 13:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4657/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 15:31:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 13: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 21:59:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 11:
LocalIncompleteTable can only be acquired by getTableIfCached which only exists on the code path of GET_TABLES HS2 operation. It won't be used at other places. As long as getTable is called, a completely loaded table will be acquired. So having LocalIncompleteTable#isLoaded return true as same as LocalFsTable does won't cause troubles in the previous patch.
Anyway, to avoid confusions or introducing bugs in the future, I fix the isLoaded implementation.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 18:18:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13874
to look at the new patch set (#11).
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES for all tables on a cold started cluster. With
this patch, the time reduces to ~1s.
- Testing in HUE-4.4.0 with a db with 3000 tables, the performance is the
same as using legacy catalog implementation.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 156 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/11
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13874
to look at the new patch set (#7).
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES on a cold started cluster. With this patch,
the time reduces to ~1s.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 153 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/7
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4636/
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 09:22:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4636/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 15:31:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has removed a vote on this change.
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Removed Code-Review-2 by Quanlong Huang <hu...@gmail.com>
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 6:
(4 comments)
Addressing the comments. Trying to test it in a live cluster with HUE.
http://gerrit.cloudera.org:8080/#/c/13874/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13874/5//COMMIT_MSG@23
PS5, Line 23: - Testing in a HMS with 100 dbs and 3000 tables, without this patch it
> Mind deploying this jar on a cluster with Hue and quickly check that nothin
Sure. Will do this.
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java
File fe/src/main/java/org/apache/impala/catalog/Catalog.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java@167
PS5, Line 167: Catalog v1
> I think the code never refers to Catalog "v1" and Catalog "v2". So this cou
I'm trying to explain why the implementation here is just delegating to getTableNoThrow. This function will only be called by impalad. Code path came here when in Catalog v1. Otherwise, the code path will go to LocalCatalog's implementation.
What about putting such kind of comments inside the function? Will it be less confusing?
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@169
PS5, Line 169: } catch (TException e) {
> Should we instead load LocalIncompleteTable here?
Cool! Done.
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/test/java/org/apache/impala/service/JdbcTest.java
File fe/src/test/java/org/apache/impala/service/JdbcTest.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/test/java/org/apache/impala/service/JdbcTest.java@390
PS5, Line 390: public void testMetaDataGetColumnComments() throws Exception {
> Mind adding a test to LocalCatalogTest#testGetTables()? Check that the tabl
Sure
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 23:23:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3899/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 14:58:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has removed a vote on this change.
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Removed Code-Review+2 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 12:
The build failure is in ubuntu-16.04-dockerised-tests Jenkins job. It failed to compile "External Data Source API":
========================================================================
Running mvn -B install -DskipTests
Directory /home/ubuntu/Impala/query-event-hook-api
========================================================================
========================================================================
Running mvn -B install -DskipTests
Directory /home/ubuntu/Impala/ext-data-source
========================================================================
========================================================================
Running mvn -B install -DskipTests
Directory /home/ubuntu/Impala/shaded-deps
========================================================================
[INFO] Apache Impala External Data Source Test Library [jar]
[INFO] Apache Impala External Data Source 1.0-SNAPSHOT .... SUCCESS [ 0.245 s]
[INFO] Apache Impala External Data Source API ............. FAILURE [ 3.537 s]
[INFO] Apache Impala External Data Source Test Library 0.1-SNAPSHOT SKIPPED
[INFO] BUILD FAILURE
[ERROR] Plugin org.apache.maven.plugins:maven-surefire-plugin:2.12.4 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.maven.plugins:maven-surefire-plugin:jar:2.12.4 from/to central (https://repo.maven.apache.org/maven2): /home/ubuntu/.m2/repository/org/apache/maven/plugins/maven-surefire-plugin/2.12.4/maven-surefire-plugin-2.12.4.jar.part (No such file or directory) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
mvn -B install -DskipTests exited with code 0
ext-data-source/CMakeFiles/ext-data-source.dir/build.make:57: recipe for target 'ext-data-source/CMakeFiles/ext-data-source' failed
make[3]: *** [ext-data-source/CMakeFiles/ext-data-source] Error 1
CMakeFiles/Makefile2:13363: recipe for target 'ext-data-source/CMakeFiles/ext-data-source.dir/all' failed
make[2]: *** [ext-data-source/CMakeFiles/ext-data-source.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
Rerun the build since it's not relative to this patch.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 15:30:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3913/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 19 Jul 2019 00:02:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 11: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Jul 2019 01:03:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10:
Hit flaky test failure as IMPALA-8378: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6721/
Re-run the GVO again.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 12:43:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 19:13:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES on a cold started cluster. With this patch,
the time reduces to ~1s.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 141 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/6
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3962/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 18:52:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 10: Code-Review-2
Found that LocalIncompleteTable#isLoaded returns true. Need to fix it and check more code paths.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 23 Jul 2019 17:48:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/3897/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Jul 2019 14:12:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13874
to look at the new patch set (#8).
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES for all tables on a cold started cluster. With
this patch, the time reduces to ~1s.
- Testing in HUE-4.4.0 with a db with 3000 tables, the performance is the
same as using legacy catalog implementation.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
11 files changed, 153 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/8
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 5:
(4 comments)
Patch makes sense to me. Some comments.
http://gerrit.cloudera.org:8080/#/c/13874/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13874/5//COMMIT_MSG@23
PS5, Line 23: - Testing in a HMS with 100 dbs and 3000 tables, without this patch it
Mind deploying this jar on a cluster with Hue and quickly check that nothing breaks?
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java
File fe/src/main/java/org/apache/impala/catalog/Catalog.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java@167
PS5, Line 167: Catalog v1
I think the code never refers to Catalog "v1" and Catalog "v2". So this could be confusing and better to get rid of them (multiple places).
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@169
PS5, Line 169: newMap.put(tableName.toLowerCase(), null);
Should we instead load LocalIncompleteTable here?
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/test/java/org/apache/impala/service/JdbcTest.java
File fe/src/test/java/org/apache/impala/service/JdbcTest.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/test/java/org/apache/impala/service/JdbcTest.java@390
PS5, Line 390: public void testMetaDataGetColumnComments() throws Exception {
Mind adding a test to LocalCatalogTest#testGetTables()? Check that the table remains in an incomplete state after GET_TABLES request?
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 18 Jul 2019 17:28:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 5:
(2 comments)
Couple of nits, but can +2 once fixed.
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java
File fe/src/main/java/org/apache/impala/catalog/Catalog.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java@167
PS5, Line 167: Catalog v1
> I'm trying to explain why the implementation here is just delegating to get
Right, I'm talking about the terminology. In the code, we never see these terms like v1 and v2. Maybe say legacy implementation vs LocalCatalog implementation or some such thing?
http://gerrit.cloudera.org:8080/#/c/13874/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
File fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java:
http://gerrit.cloudera.org:8080/#/c/13874/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@385
PS6, Line 385:
Why this
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 19 Jul 2019 16:16:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13874
to look at the new patch set (#5).
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
In LocalCatalog implementation, LocalDb#getTable will always return a
completely loaded table containing all the meta of columns, partitions,
files, etc. It's time consuming if we implement the GET_TABLES
HiveServer2 operation based on this interface, since GET_TABLES only
requires table names, table types and table comments, while this
interface will trigger catalogd to fully load the table meta. It becomes
worse when we do this for all the tables.
This patch introduces a new interface, getTableIfCached, to return a
LocalIncompleteTable object if the corresponding table is unloaded,
which requires no round trips to the catalogd. It's used to boost the
GET_TABLES performance in LocalCatalog mode.
Tests
- Testing in a HMS with 100 dbs and 3000 tables, without this patch it
takes ~2mins in GET_TABLES on a cold started cluster. With this patch,
the time reduces to ~1s.
Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
---
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalIncompleteTable.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M tests/hs2/test_hs2.py
10 files changed, 122 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13874/5
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode
......................................................................
Patch Set 8:
There're some bugs in the current master branch of HUE that not showing anything in the table browser tab. I managed to test with HUE in version 4.4.0-release. Everything is fine :)
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Mon, 22 Jul 2019 17:28:09 +0000
Gerrit-HasComments: No