You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "wecharyu (via GitHub)" <gi...@apache.org> on 2024/05/31 03:58:40 UTC
[PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
wecharyu opened a new pull request, #5273:
URL: https://github.com/apache/hive/pull/5273
### What changes were proposed in this pull request?
Make some improvements in `SHOW TABLES` statement:
1. Push down table pattern to metastore
2. Request table names instead of table objects
3. Change index `UNIQUETABLE(TBL_NAME, DB_ID)` to `UNIQUETABLE (DB_ID, TBL_NAME)` and drop index `TBLS_N49(DB_ID)`
### Why are the changes needed?
Improve the performance of `SHOW TABLES|VIEWS` statements.
### Does this PR introduce _any_ user-facing change?
No.
### Is the change a dependency upgrade?
No.
### How was this patch tested?
Passing existing tests and do benchmark tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2351095781
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5273&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5273&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626142935
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf) {
public abstract void showTables(DataOutputStream out, List<String> tables) throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, List<Pair<String, String>> tables) throws HiveException;
Review Comment:
why is that change? if you'd like to avoid using fat class, maybe try org.apache.hadoop.hive.metastore.api.Table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626176204
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
Review Comment:
is that an extra db call? what do we save here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626310126
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##########
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
- List<Table> tableObjects = new ArrayList<>();
- tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(), null, desc.getTypeFilter()));
- if (desc.getPattern() != null) {
- Pattern pattern = Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()), Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+ List<Pair<String, String>> tableTypePairs = new ArrayList<>();
+ String pattern = MetaStoreUtils.convertSqlPatternToRegExp(desc.getPattern());
+ TableType typeFilter = desc.getTypeFilter();
+ TableType[] tableTypes = typeFilter == null ? TableType.values() : new TableType[]{typeFilter};
Review Comment:
It means to request all types table names if not input type filter in statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626154307
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1315,4 +1316,35 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ public static String convertSqlPatternToRegExp(String likePattern) {
Review Comment:
it looks like a copy&paste from `UDFLike.likePatternToRegExp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626324311
##########
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.1.0.derby.sql:
##########
@@ -269,7 +269,7 @@ CREATE UNIQUE INDEX "APP"."ROLEENTITYINDEX" ON "APP"."ROLES" ("ROLE_NAME");
CREATE INDEX "APP"."TABLEPRIVILEGEINDEX" ON "APP"."TBL_PRIVS" ("AUTHORIZER", "TBL_ID", "PRINCIPAL_NAME", "PRINCIPAL_TYPE", "TBL_PRIV", "GRANTOR", "GRANTOR_TYPE");
-CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("TBL_NAME", "DB_ID");
+CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("DB_ID", "TBL_NAME");
Review Comment:
We change the order of this composite index to improve performance. Because we always get the`DB_ID` first while doing any query for hive table, so the first level (DB_ID) of the index alway take effect after this change.
It's a similar change like https://github.com/apache/hive/pull/4831#pullrequestreview-1814098075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626182898
##########
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.1.0.derby.sql:
##########
@@ -269,7 +269,7 @@ CREATE UNIQUE INDEX "APP"."ROLEENTITYINDEX" ON "APP"."ROLES" ("ROLE_NAME");
CREATE INDEX "APP"."TABLEPRIVILEGEINDEX" ON "APP"."TBL_PRIVS" ("AUTHORIZER", "TBL_ID", "PRINCIPAL_NAME", "PRINCIPAL_TYPE", "TBL_PRIV", "GRANTOR", "GRANTOR_TYPE");
-CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("TBL_NAME", "DB_ID");
+CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("DB_ID", "TBL_NAME");
Review Comment:
what does it change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1760046167
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -77,9 +80,9 @@ public static String likePatternToRegExp(String likePattern) {
if (n == '_') {
sb.append(".");
} else if (n == '%') {
- sb.append(".*?");
+ sb.append(".*");
Review Comment:
JDO does not support non-greedy matching :(
Add a new parameter `greedyMatch` for this option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759696055
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java:
##########
@@ -321,7 +321,7 @@ public List<String> getTables(String dbName, String tablePattern) throws MetaExc
if (tables == null || tables.size() == 0) {
return tableNames;
}
- tablePattern = tablePattern.replaceAll("\\*", ".*");
+ tablePattern = tablePattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
Review Comment:
@wecharyu, could we replace it with `tablePattern.replaceAll("(?<!\\.)\\*", ".*")` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759773049
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -61,7 +61,10 @@ private enum PatternType {
public UDFLike() {
}
- public static String likePatternToRegExp(String likePattern) {
+ public static String likePatternToRegExp(String likePattern, boolean quote) {
Review Comment:
Addressed.
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -77,9 +80,9 @@ public static String likePatternToRegExp(String likePattern) {
if (n == '_') {
sb.append(".");
} else if (n == '%') {
- sb.append(".*?");
+ sb.append(".*");
Review Comment:
Reverted. We only use the pattern for string matching, is reasonable to use the non-greedy matching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759782339
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -96,8 +97,8 @@ public void showTables(DataOutputStream out, List<String> tables) throws HiveExc
}
@Override
- void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException {
- if (tables.isEmpty()) {
+ void showTablesExtended(DataOutputStream out, TreeMap<String, String> tableNameToType) throws HiveException {
Review Comment:
`Map`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2141648245
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2143588777
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2351622001
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5273&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5273&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759782304
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -62,16 +63,16 @@ public void showTables(DataOutputStream out, List<String> tables) throws HiveExc
}
@Override
- void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException {
- if (tables.isEmpty()) {
+ void showTablesExtended(DataOutputStream out, TreeMap<String, String> tableNameToType) throws HiveException {
Review Comment:
`Map`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1657061676
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
+ } catch (Exception e) { /* appears exception is not thrown currently if db doesnt exist */ }
+
+ if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+ req.setProcessorCapabilities(Arrays.asList(processorCapabilities));
+ if (processorIdentifier != null)
+ req.setProcessorIdentifier(processorIdentifier);
+ req.setProjectionSpec(projectionsSpec);
+ List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
+ tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
+ for (Table tbl : tableObjects) {
+ tables.add(tbl.getTableName());
+ }
+ } else {
+ // This trick handles pattern for both string regex and wildcards ('*' and '|').
+ // We need unify the pattern definition, see HIVE-28297 for details.
+ String[] patterns = tablePattern.split("\\|");
+ for (String pattern : patterns) {
+ pattern = "(?i)" + pattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
Review Comment:
what is `(?i)` used for and 2 replaceAll?
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
+ } catch (Exception e) { /* appears exception is not thrown currently if db doesnt exist */ }
+
+ if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+ req.setProcessorCapabilities(Arrays.asList(processorCapabilities));
+ if (processorIdentifier != null)
+ req.setProcessorIdentifier(processorIdentifier);
+ req.setProjectionSpec(projectionsSpec);
+ List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
+ tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
+ for (Table tbl : tableObjects) {
+ tables.add(tbl.getTableName());
+ }
+ } else {
+ // This trick handles pattern for both string regex and wildcards ('*' and '|').
+ // We need unify the pattern definition, see HIVE-28297 for details.
+ String[] patterns = tablePattern.split("\\|");
+ for (String pattern : patterns) {
+ pattern = "(?i)" + pattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
+ String filter = hive_metastoreConstants.HIVE_FILTER_FIELD_TABLE_NAME + " like \"" + pattern + "\"";
Review Comment:
why not use String.format("like \"%s\"", hive_metastoreConstants.HIVE_FILTER_FIELD_TABLE_NAM)
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
+ } catch (Exception e) { /* appears exception is not thrown currently if db doesnt exist */ }
+
+ if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+ req.setProcessorCapabilities(Arrays.asList(processorCapabilities));
+ if (processorIdentifier != null)
+ req.setProcessorIdentifier(processorIdentifier);
+ req.setProjectionSpec(projectionsSpec);
+ List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
+ tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
+ for (Table tbl : tableObjects) {
+ tables.add(tbl.getTableName());
+ }
+ } else {
+ // This trick handles pattern for both string regex and wildcards ('*' and '|').
+ // We need unify the pattern definition, see HIVE-28297 for details.
+ String[] patterns = tablePattern.split("\\|");
+ for (String pattern : patterns) {
+ pattern = "(?i)" + pattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
+ String filter = hive_metastoreConstants.HIVE_FILTER_FIELD_TABLE_NAME + " like \"" + pattern + "\"";
Review Comment:
why not use `String.format("like \"%s\"", hive_metastoreConstants.HIVE_FILTER_FIELD_TABLE_NAM)`
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1315,4 +1316,35 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ public static String convertSqlPatternToRegExp(String likePattern) {
Review Comment:
ok, why not to add param `bool quote` and reuse the method
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1315,4 +1316,35 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ public static String convertSqlPatternToRegExp(String likePattern) {
Review Comment:
ok, why not add param `bool quote` and reuse the method
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -4941,6 +4929,14 @@ private Table ensureGetTable(String catName, String dbName, String tblName)
return convertToTable(ensureGetMTable(catName, dbName, tblName));
}
+ private Database ensureGetDatabase(String catName, String dbName) throws UnknownDBException {
Review Comment:
is that used in multiple places? if not just inline into method where it's used
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -4941,6 +4929,14 @@ private Table ensureGetTable(String catName, String dbName, String tblName)
return convertToTable(ensureGetMTable(catName, dbName, tblName));
}
+ private Database ensureGetDatabase(String catName, String dbName) throws UnknownDBException {
Review Comment:
is that used in multiple places? if not just inline into method where it's used
##########
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.1.0.derby.sql:
##########
@@ -269,7 +269,7 @@ CREATE UNIQUE INDEX "APP"."ROLEENTITYINDEX" ON "APP"."ROLES" ("ROLE_NAME");
CREATE INDEX "APP"."TABLEPRIVILEGEINDEX" ON "APP"."TBL_PRIVS" ("AUTHORIZER", "TBL_ID", "PRINCIPAL_NAME", "PRINCIPAL_TYPE", "TBL_PRIV", "GRANTOR", "GRANTOR_TYPE");
-CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("TBL_NAME", "DB_ID");
+CREATE UNIQUE INDEX "APP"."UNIQUETABLE" ON "APP"."TBLS" ("DB_ID", "TBL_NAME");
Review Comment:
are you saying we won't hit the index if the order of columns is reversed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626133900
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf) {
public abstract void showTables(DataOutputStream out, List<String> tables) throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, List<Pair<String, String>> tables) throws HiveException;
Review Comment:
wouldn't it be better to use `TableName`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626142935
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf) {
public abstract void showTables(DataOutputStream out, List<String> tables) throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, List<Pair<String, String>> tables) throws HiveException;
Review Comment:
why is that change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2146520644
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759697135
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
+ } catch (Exception e) { /* appears exception is not thrown currently if db doesnt exist */ }
+
+ if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+ req.setProcessorCapabilities(Arrays.asList(processorCapabilities));
+ if (processorIdentifier != null)
+ req.setProcessorIdentifier(processorIdentifier);
+ req.setProjectionSpec(projectionsSpec);
+ List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
+ tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
+ for (Table tbl : tableObjects) {
+ tables.add(tbl.getTableName());
+ }
+ } else {
+ // This trick handles pattern for both string regex and wildcards ('*' and '|').
+ // We need unify the pattern definition, see HIVE-28297 for details.
+ String[] patterns = tablePattern.split("\\|");
+ for (String pattern : patterns) {
+ pattern = "(?i)" + pattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
Review Comment:
could we use negative lookbehind here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759696055
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java:
##########
@@ -321,7 +321,7 @@ public List<String> getTables(String dbName, String tablePattern) throws MetaExc
if (tables == null || tables.size() == 0) {
return tableNames;
}
- tablePattern = tablePattern.replaceAll("\\*", ".*");
+ tablePattern = tablePattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
Review Comment:
@wecharyu, could we replace the pattern with the negative lookbehind like `tablePattern.replaceAll("(?<!\\.)\\*", ".*")` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759699154
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -77,9 +80,9 @@ public static String likePatternToRegExp(String likePattern) {
if (n == '_') {
sb.append(".");
} else if (n == '%') {
- sb.append(".*?");
+ sb.append(".*");
Review Comment:
I am not sure about this change, should we keep the matching lazy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2146451098
@dengzhhu653 @deniskuzZ @zhangbutao Could you please help review this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626287681
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf) {
public abstract void showTables(DataOutputStream out, List<String> tables) throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, List<Pair<String, String>> tables) throws HiveException;
Review Comment:
`SHOW TABLES EXTENDED` only outputs table name and table type, so we do not need the `Table` objects here.
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
##########
@@ -1315,4 +1316,35 @@ public static String getHttpPath(String httpPath) {
}
return httpPath;
}
+
+ public static String convertSqlPatternToRegExp(String likePattern) {
Review Comment:
Yes, but `UDFLike.likePatternToRegExp` is used for `java.util.regex.Pattern`, so it quote the characters which is not expected in jdo String matches.
https://github.com/apache/hive/blob/e570e65a397aae47c62472f12acaaed06b68cf54/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java#L77-L82
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626164009
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##########
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
- List<Table> tableObjects = new ArrayList<>();
- tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(), null, desc.getTypeFilter()));
- if (desc.getPattern() != null) {
- Pattern pattern = Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()), Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+ List<Pair<String, String>> tableTypePairs = new ArrayList<>();
+ String pattern = MetaStoreUtils.convertSqlPatternToRegExp(desc.getPattern());
+ TableType typeFilter = desc.getTypeFilter();
+ TableType[] tableTypes = typeFilter == null ? TableType.values() : new TableType[]{typeFilter};
Review Comment:
why is this magic with typeFilter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626162410
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##########
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
- List<Table> tableObjects = new ArrayList<>();
- tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(), null, desc.getTypeFilter()));
- if (desc.getPattern() != null) {
- Pattern pattern = Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()), Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+ List<Pair<String, String>> tableTypePairs = new ArrayList<>();
Review Comment:
can't you use Map here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2145970047
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [12 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759700364
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -61,7 +61,10 @@ private enum PatternType {
public UDFLike() {
}
- public static String likePatternToRegExp(String likePattern) {
+ public static String likePatternToRegExp(String likePattern, boolean quote) {
Review Comment:
since it's used in multiple places maybe you can keep the default impl with 1 param
````
public static String likePatternToRegExp(likePattern){
return likePatternToRegExp(likePattern, true);
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759782378
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##########
@@ -78,20 +73,19 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
- List<Table> tableObjects = new ArrayList<>();
- tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(), null, desc.getTypeFilter()));
- if (desc.getPattern() != null) {
- Pattern pattern = Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()), Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+ TreeMap<String, String> tableNameToType = new TreeMap<>();
Review Comment:
`Map`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759696055
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java:
##########
@@ -321,7 +321,7 @@ public List<String> getTables(String dbName, String tablePattern) throws MetaExc
if (tables == null || tables.size() == 0) {
return tableNames;
}
- tablePattern = tablePattern.replaceAll("\\*", ".*");
+ tablePattern = tablePattern.replaceAll("\\.\\*", "\\*").replaceAll("\\*", ".*");
Review Comment:
@wecharyu, could we replace this with `tablePattern.replaceAll("(?<!\\.)\\*", ".*")` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759782217
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf) {
public abstract void showTables(DataOutputStream out, List<String> tables) throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, TreeMap<String, String> tableNameToType) throws HiveException;
Review Comment:
please code to an interface, use `Map`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1623274038
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesGetExists.java:
##########
@@ -286,13 +286,6 @@ public void testGetTables() throws Exception {
Assert.assertTrue("Comparing tablenames", tables.contains(testTables[5].getTableName()));
Assert.assertTrue("Comparing tablenames", tables.contains(testTables[6].getTableName()));
- tables = client.getTables("*", "*");
- Assert.assertEquals("All tables size", 7, tables.size());
- tables = client.getTables("d*", "*");
- Assert.assertEquals("All tables size", 7, tables.size());
- tables = client.getTables("def*", "*");
- Assert.assertEquals("All tables size", 5, tables.size());
Review Comment:
Remove these tests, discussed in https://github.com/apache/hive/pull/4326#issuecomment-2143551034.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626133900
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesFormatter.java:
##########
@@ -51,7 +52,7 @@ public static ShowTablesFormatter getFormatter(HiveConf conf) {
public abstract void showTables(DataOutputStream out, List<String> tables) throws HiveException;
- abstract void showTablesExtended(DataOutputStream out, List<Table> tables) throws HiveException;
+ abstract void showTablesExtended(DataOutputStream out, List<Pair<String, String>> tables) throws HiveException;
Review Comment:
wouldn't it be better to use `TableName`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626333863
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##########
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
- List<Table> tableObjects = new ArrayList<>();
- tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(), null, desc.getTypeFilter()));
- if (desc.getPattern() != null) {
- Pattern pattern = Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()), Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+ List<Pair<String, String>> tableTypePairs = new ArrayList<>();
Review Comment:
Good idea, addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626187357
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
+ } catch (Exception e) { /* appears exception is not thrown currently if db doesnt exist */ }
+
+ if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+ req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
Review Comment:
what is that double ArrayList?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626162410
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java:
##########
@@ -78,20 +75,22 @@ private void showTables() throws HiveException {
}
private void showTablesExtended() throws HiveException {
- List<Table> tableObjects = new ArrayList<>();
- tableObjects.addAll(context.getDb().getTableObjects(desc.getDbName(), null, desc.getTypeFilter()));
- if (desc.getPattern() != null) {
- Pattern pattern = Pattern.compile(UDFLike.likePatternToRegExp(desc.getPattern()), Pattern.CASE_INSENSITIVE);
- tableObjects = tableObjects.stream()
- .filter(object -> pattern.matcher(object.getTableName()).matches())
- .collect(Collectors.toList());
+ List<Pair<String, String>> tableTypePairs = new ArrayList<>();
Review Comment:
can't you use TreeMap here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626171883
##########
ql/src/test/results/clientpositive/llap/authorization_privilege_objects.q.out:
##########
@@ -42,8 +42,8 @@ PREHOOK: query: SHOW TABLES IN test_auth_obj_db
PREHOOK: type: SHOWTABLES
PREHOOK: Input: database:test_auth_obj_db
filterListCmdObjects
-HIVE PRIVILEGE OBJECT { objectName: test_privs type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db OWNER: testuser OWNERTYPE: USER}
-HIVE PRIVILEGE OBJECT { objectName: test_privs2 type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db OWNER: testuser2 OWNERTYPE: USER}
+HIVE PRIVILEGE OBJECT { objectName: test_privs type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db}
+HIVE PRIVILEGE OBJECT { objectName: test_privs2 type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db}
Review Comment:
why owner information is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #5273:
URL: https://github.com/apache/hive/pull/5273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759699154
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -77,9 +80,9 @@ public static String likePatternToRegExp(String likePattern) {
if (n == '_') {
sb.append(".");
} else if (n == '%') {
- sb.append(".*?");
+ sb.append(".*");
Review Comment:
I am not sure about this change, should we keep the matching lazy? see HIVE-13196 : UDFLike: reduce Regex NFA sizes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1759701400
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFLike.java:
##########
@@ -61,7 +61,10 @@ private enum PatternType {
public UDFLike() {
}
- public static String likePatternToRegExp(String likePattern) {
+ public static String likePatternToRegExp(String likePattern, boolean quote) {
Review Comment:
maybe `literalize` instead of `quote`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626328981
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
+ } catch (Exception e) { /* appears exception is not thrown currently if db doesnt exist */ }
+
+ if (MetaStoreUtils.isDatabaseRemote(db)) {
+ // TODO: remote database does not support list table names by pattern yet.
+ // This branch can be removed once it's supported.
+ GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
+ projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
+ GetTablesRequest req = new GetTablesRequest(dbName);
+ req.setCatName(catName);
+ req.setCapabilities(version);
+ req.setTblNames(null);
+ req.setTablesPattern(tablePattern);
+ if (processorCapabilities != null)
+ req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
Review Comment:
It's copied from source, removed redundant ArrayList.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5273:
URL: https://github.com/apache/hive/pull/5273#issuecomment-2148093437
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273) **Quality Gate passed**
Issues
 [39 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5273&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5273&resolved=false&sinceLeakPeriod=true)
 No data about Coverage
 No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5273)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626315585
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
##########
@@ -2897,26 +2897,44 @@ public List<String> getTables(String dbname, String tablePattern) throws MetaExc
@Override
public List<String> getTables(String catName, String dbName, String tablePattern)
throws TException {
- List<String> tables = new ArrayList<>();
- GetProjectionsSpec projectionsSpec = new GetProjectionsSpec();
- projectionsSpec.setFieldList(Arrays.asList("dbName", "tableName", "owner", "ownerType"));
- GetTablesRequest req = new GetTablesRequest(dbName);
- req.setCatName(catName);
- req.setCapabilities(version);
- req.setTblNames(null);
- if(tablePattern == null){
+ if (tablePattern == null) {
tablePattern = ".*";
}
- req.setTablesPattern(tablePattern);
- if (processorCapabilities != null)
- req.setProcessorCapabilities(new ArrayList<String>(Arrays.asList(processorCapabilities)));
- if (processorIdentifier != null)
- req.setProcessorIdentifier(processorIdentifier);
- req.setProjectionSpec(projectionsSpec);
- List<Table> tableObjects = client.get_table_objects_by_name_req(req).getTables();
- tableObjects = deepCopyTables(FilterUtils.filterTablesIfEnabled(isClientFilterEnabled, filterHook, tableObjects));
- for (Table tbl : tableObjects) {
- tables.add(tbl.getTableName());
+ List<String> tables = new ArrayList<>();
+ Database db = null;
+ try {
+ db = getDatabase(catName, dbName);
Review Comment:
Yes, we need it to determine if this is a remote database which does not support table pattern yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28292: Optimize SHOW TABLES|VIEWS statements [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #5273:
URL: https://github.com/apache/hive/pull/5273#discussion_r1626312854
##########
ql/src/test/results/clientpositive/llap/authorization_privilege_objects.q.out:
##########
@@ -42,8 +42,8 @@ PREHOOK: query: SHOW TABLES IN test_auth_obj_db
PREHOOK: type: SHOWTABLES
PREHOOK: Input: database:test_auth_obj_db
filterListCmdObjects
-HIVE PRIVILEGE OBJECT { objectName: test_privs type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db OWNER: testuser OWNERTYPE: USER}
-HIVE PRIVILEGE OBJECT { objectName: test_privs2 type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db OWNER: testuser2 OWNERTYPE: USER}
+HIVE PRIVILEGE OBJECT { objectName: test_privs type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db}
+HIVE PRIVILEGE OBJECT { objectName: test_privs2 type: TABLE_OR_VIEW actionType: OTHER dbName: test_auth_obj_db}
Review Comment:
since we turns to request only table names instead of table objects, so here auth check will not need owner info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org