You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Aman Sinha (Code Review)" <ge...@cloudera.org> on 2020/04/05 19:30:04 UTC
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Aman Sinha has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15653
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
IMPALA-9602: Fix case-sensitivity for local catalog
This patch makes the database and table names lower-case
when doing lookups, insertion and invalidations of database
objects or table objects in the local catalog cache. The
remote catalog already does the right thing by lower-casing
these names, so this patch makes the behavior consistent with
what the remote catalog does.
Testing:
- Added unit tests for CatalogdMetaProvider by examining
cache hits and misses when loading and invalidating
database or tables with upper-case names.
- Manually tested as follows:
start Impala with local catalog enabled
start-impala-cluster.py
--catalogd_args="--catalog_topic_mode=minimal"
--impalad_args="--use_local_catalog=true"
Create database in lower-case: "CREATE DATABASE db1;"
Run the following a few times (this errors without the patch):
impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE;
CREATE TABLE DB1.ddl_test1 (val string);"
Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
---
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
2 files changed, 64 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/15653/1
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5701/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Apr 2020 20:11:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5627/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2020 18:10:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG@21
PS3, Line 21: start Impala with local catalog enabled
: start-impala-cluster.py
: --catalogd_args="--catalog_topic_mode=minimal"
: --impalad_args="--use_local_catalog=true"
: Create database in lower-case: "CREATE DATABASE db1;"
: Run the following a few times (this errors without the patch):
: impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE;
: CREATE TABLE DB1.ddl_test1 (val string)
> We have a test_local_catalog.py or test_ddl.py in case you want to add this
That's good to know about the python script which seems great for end-to-end test and testing race conditions in cache updates. I will keep this in mind for future testing. In this case though, since the problem is simpler, I am thinking the unit test is good enough. Let me know if you think otherwise.
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@625
PS3, Line 625: DbCacheKey
> May be its better to move toLowerCase() call inside DbCacheKey constructor.
I had thought of that but looking at existing code in other places (e.g Catalog.java, CatalogObjectCache.java), the caller explicitly lowercases instead of the constructor doing it. One motivation for it is there's a caseInsensitiveKeys_ flag in CatalogObjectCache to do this conditionally. I am thinking it would be good to maintain similar logic in the local catalog in case in the future we want to do this conditionally for any reason. What do you think ?
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2020 01:31:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15653
to look at the new patch set (#3).
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
IMPALA-9602: Fix case-sensitivity for local catalog
This patch makes the database and table names lower-case
when doing lookups, insertion and invalidations of database
objects or table objects in the local catalog cache. The
remote catalog already does the right thing by lower-casing
these names, so this patch makes the behavior consistent with
what the remote catalog does.
Testing:
- Added unit tests for CatalogdMetaProvider by examining
cache hits and misses when loading and invalidating
database or tables with upper-case names.
- Manually tested as follows:
start Impala with local catalog enabled
start-impala-cluster.py
--catalogd_args="--catalog_topic_mode=minimal"
--impalad_args="--use_local_catalog=true"
Create database in lower-case: "CREATE DATABASE db1;"
Run the following a few times (this errors without the patch):
impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE;
CREATE TABLE DB1.ddl_test1 (val string);"
Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
---
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
2 files changed, 65 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/15653/3
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5713/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Apr 2020 02:28:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5702/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Apr 2020 22:36:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5610/
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Apr 2020 02:44:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@681
PS1, Line 681: TableCacheKey cacheKey = new TableCacheKey(dbName.toLowerCase(), tableName.toLowerCase());
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java:
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@340
PS1, Line 340: // names as case-insensitive.
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Apr 2020 21:55:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5632/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Apr 2020 00:13:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 4:
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5627/
It looks like there is some kind of mini-cluster setup issue. java.lang.IllegalStateException: Failed to assign regions to servers after 60000 millis
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Apr 2020 00:12:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@681
PS1, Line 681: TableCacheKey cacheKey = new TableCacheKey(dbName.toLowerCase(), tableName.toLowerCase());
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
File fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java:
http://gerrit.cloudera.org:8080/#/c/15653/1/fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java@340
PS1, Line 340: // names as case-insensitive.
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 1
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Apr 2020 19:31:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Apr 2020 04:46:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 5
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Apr 2020 00:13:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2020 18:10:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 3: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG@21
PS3, Line 21: start Impala with local catalog enabled
: start-impala-cluster.py
: --catalogd_args="--catalog_topic_mode=minimal"
: --impalad_args="--use_local_catalog=true"
: Create database in lower-case: "CREATE DATABASE db1;"
: Run the following a few times (this errors without the patch):
: impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE;
: CREATE TABLE DB1.ddl_test1 (val string)
> That's good to know about the python script which seems great for end-to-en
sure. Since the existing unit-test will mostly be doing the same I think its okay for now.
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@625
PS3, Line 625: DbCacheKey
> I had thought of that but looking at existing code in other places (e.g Cat
yeah, I am okay with that.
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2020 01:56:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5627/
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 4
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2020 22:37:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5610/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Apr 2020 22:17:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
IMPALA-9602: Fix case-sensitivity for local catalog
This patch makes the database and table names lower-case
when doing lookups, insertion and invalidations of database
objects or table objects in the local catalog cache. The
remote catalog already does the right thing by lower-casing
these names, so this patch makes the behavior consistent with
what the remote catalog does.
Testing:
- Added unit tests for CatalogdMetaProvider by examining
cache hits and misses when loading and invalidating
database or tables with upper-case names.
- Manually tested as follows:
start Impala with local catalog enabled
start-impala-cluster.py
--catalogd_args="--catalog_topic_mode=minimal"
--impalad_args="--use_local_catalog=true"
Create database in lower-case: "CREATE DATABASE db1;"
Run the following a few times (this errors without the patch):
impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE;
CREATE TABLE DB1.ddl_test1 (val string);"
Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Reviewed-on: http://gerrit.cloudera.org:8080/15653
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
2 files changed, 65 insertions(+), 8 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 6
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
[Impala-ASF-CR] IMPALA-9602: Fix case-sensitivity for local catalog
Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15653 )
Change subject: IMPALA-9602: Fix case-sensitivity for local catalog
......................................................................
Patch Set 3:
(3 comments)
Patch looks good to me. Left a few suggestions below.
http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15653/3//COMMIT_MSG@21
PS3, Line 21: start Impala with local catalog enabled
: start-impala-cluster.py
: --catalogd_args="--catalog_topic_mode=minimal"
: --impalad_args="--use_local_catalog=true"
: Create database in lower-case: "CREATE DATABASE db1;"
: Run the following a few times (this errors without the patch):
: impala-shell.sh -q "DROP TABLE IF EXISTS DB1.ddl_test1 PURGE;
: CREATE TABLE DB1.ddl_test1 (val string)
We have a test_local_catalog.py or test_ddl.py in case you want to add this test case.
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@625
PS3, Line 625: DbCacheKey
May be its better to move toLowerCase() call inside DbCacheKey constructor. That way any future usage of dbName will not need a explicit lower-casing.
http://gerrit.cloudera.org:8080/#/c/15653/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@681
PS3, Line 681: TableCacheKey
same as earlier, may be moving to TableCacheKey constructor is better.
--
To view, visit http://gerrit.cloudera.org:8080/15653
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f368fa9b50e22ec5057d0bf66c3fd51064d4c26
Gerrit-Change-Number: 15653
Gerrit-PatchSet: 3
Gerrit-Owner: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Apr 2020 18:19:05 +0000
Gerrit-HasComments: Yes