You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/10/25 09:20:04 UTC
[GitHub] [hive] ashish-kumar-sharma opened a new pull request #1610: HIVE-24259: [CachedStore] Optimise getAlltableConstraint from 6 cache call to 1 cache call
ashish-kumar-sharma opened a new pull request #1610:
URL: https://github.com/apache/hive/pull/1610
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r575929235
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -506,10 +512,18 @@ boolean cacheCheckConstraints(List<SQLCheckConstraint> checkConstraints, boolean
return cacheConstraints(checkConstraints, fromPrewarm, MemberName.CHECK_CONSTRAINT_CACHE);
}
+ boolean cacheConstraints(SQLAllTableConstraints constraints, boolean fromPrewarm) {
+ this.isConstraintsValid =
+ cacheConstraints(constraints.getPrimaryKeys(), fromPrewarm, MemberName.PRIMARY_KEY_CACHE) && cacheConstraints(
+ constraints.getForeignKeys(), fromPrewarm, MemberName.FOREIGN_KEY_CACHE) && cacheConstraints(constraints.getDefaultConstraints(), fromPrewarm, MemberName.DEFAULT_CONSTRAINT_CACHE)
+ && cacheConstraints(constraints.getUniqueConstraints(), fromPrewarm, MemberName.UNIQUE_CONSTRAINT_CACHE)
+ && cacheConstraints(constraints.getNotNullConstraints(), fromPrewarm, MemberName.NOTNULL_CONSTRAINT_CACHE)
+ && cacheConstraints(constraints.getCheckConstraints(), fromPrewarm, MemberName.CHECK_CONSTRAINT_CACHE);
+ return this.isConstraintsValid;
+ }
+
// Common method to cache constraints
- private boolean cacheConstraints(List constraintsList,
- boolean fromPrewarm,
- MemberName mn) {
+ private boolean cacheConstraints(List constraintsList, boolean fromPrewarm, MemberName mn) {
Review comment:
In case of wrong member type.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r575038878
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -715,154 +747,105 @@ public void removeConstraint(String name) {
}
}
- public void refreshPrimaryKeys(List<SQLPrimaryKey> keys) {
- Map<String, SQLPrimaryKey> newKeys = new ConcurrentHashMap<>();
+ public void refreshAllTableConstraints(SQLAllTableConstraints constraints) {
try {
tableLock.writeLock().lock();
- int size = 0;
- for (SQLPrimaryKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, true, false)) {
- LOG.debug("Skipping primary key cache update for table: " + getTable().getTableName()
- + "; the primary keys are already refreshed.");
- return;
- }
- String pkName = StringUtils.normalizeIdentifier(key.getPk_name());
- key.setPk_name(pkName);
- newKeys.put(pkName, key);
- size += getObjectSize(SQLPrimaryKey.class, key);
- }
- primaryKeyCache = newKeys;
- updateMemberSize(MemberName.PRIMARY_KEY_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Primary keys refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
+ this.isConstraintsValid =
+ refreshConstraint(constraints.getPrimaryKeys(), MemberName.PRIMARY_KEY_CACHE) && refreshConstraint(
+ constraints.getForeignKeys(), MemberName.FOREIGN_KEY_CACHE) && refreshConstraint(
+ constraints.getUniqueConstraints(), MemberName.UNIQUE_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getDefaultConstraints(), MemberName.DEFAULT_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getNotNullConstraints(), MemberName.NOTNULL_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getCheckConstraints(), MemberName.CHECK_CONSTRAINT_CACHE);
} finally {
tableLock.writeLock().unlock();
}
}
- public void refreshForeignKeys(List<SQLForeignKey> keys) {
- Map<String, SQLForeignKey> newKeys = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLForeignKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.FOREIGN_KEY_CACHE, true, false)) {
- LOG.debug("Skipping foreign key cache update for table: " + getTable().getTableName()
- + "; the foreign keys are already refreshed.");
- return;
+ private boolean refreshConstraint(List constraints, MemberName mn) {
+ int size = 0;
+ switch (mn) {
+ case PRIMARY_KEY_CACHE:
+ Map<String, SQLPrimaryKey> newPk = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLPrimaryKey key : (List<SQLPrimaryKey>) constraints) {
+ String pkName = StringUtils.normalizeIdentifier(key.getPk_name());
+ key.setPk_name(pkName);
+ newPk.put(pkName, key);
+ size += getObjectSize(SQLPrimaryKey.class, key);
}
- String fkName = StringUtils.normalizeIdentifier(key.getFk_name());
- key.setFk_name(fkName);
- newKeys.put(fkName, key);
- size += getObjectSize(SQLForeignKey.class, key);
}
- foreignKeyCache = newKeys;
- updateMemberSize(MemberName.FOREIGN_KEY_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Foreign keys refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshNotNullConstraints(List<SQLNotNullConstraint> constraints) {
- Map<String, SQLNotNullConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLNotNullConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.NOTNULL_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping not null constraints cache update for table: " + getTable().getTableName()
- + "; the not null constraints are already refreshed.");
- return;
+ primaryKeyCache = newPk;
+ break;
+ case FOREIGN_KEY_CACHE:
+ Map<String, SQLForeignKey> newFk = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLForeignKey key : (List<SQLForeignKey>) constraints) {
+ String fkName = StringUtils.normalizeIdentifier(key.getFk_name());
+ key.setFk_name(fkName);
+ newFk.put(fkName, key);
+ size += getObjectSize(SQLForeignKey.class, key);
}
- String nnName = StringUtils.normalizeIdentifier(constraint.getNn_name());
- constraint.setNn_name(nnName);
- newConstraints.put(nnName, constraint);
- size += getObjectSize(SQLNotNullConstraint.class, constraint);
}
- notNullConstraintCache = newConstraints;
- updateMemberSize(MemberName.NOTNULL_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Not null constraints refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshUniqueConstraints(List<SQLUniqueConstraint> constraints) {
- Map<String, SQLUniqueConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLUniqueConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.UNIQUE_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping unique constraints cache update for table: " + getTable().getTableName()
- + "; the unique constraints are already refreshed.");
- return;
+ foreignKeyCache = newFk;
+ break;
+ case UNIQUE_CONSTRAINT_CACHE:
+ Map<String, SQLUniqueConstraint> newUc = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLUniqueConstraint constraint : (List<SQLUniqueConstraint>) constraints) {
+ String ucName = StringUtils.normalizeIdentifier(constraint.getUk_name());
+ constraint.setUk_name(ucName);
+ newUc.put(ucName, constraint);
+ size += getObjectSize(SQLUniqueConstraint.class, constraint);
}
- String ucName = StringUtils.normalizeIdentifier(constraint.getUk_name());
- constraint.setUk_name(ucName);
- newConstraints.put(ucName, constraint);
- size += getObjectSize(SQLUniqueConstraint.class, constraint);
}
- uniqueConstraintCache = newConstraints;
- updateMemberSize(MemberName.UNIQUE_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Unique constraints refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshDefaultConstraints(List<SQLDefaultConstraint> constraints) {
- Map<String, SQLDefaultConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLDefaultConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping default constraint cache update for table: " + getTable().getTableName()
- + "; the default constraint are already refreshed.");
- return;
+ uniqueConstraintCache = newUc;
+ break;
+ case NOTNULL_CONSTRAINT_CACHE:
+ Map<String, SQLNotNullConstraint> newNn = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLNotNullConstraint constraint : (List<SQLNotNullConstraint>) constraints) {
+ String nnName = StringUtils.normalizeIdentifier(constraint.getNn_name());
+ constraint.setNn_name(nnName);
+ newNn.put(nnName, constraint);
+ size += getObjectSize(SQLNotNullConstraint.class, constraint);
}
- String dcName = StringUtils.normalizeIdentifier(constraint.getDc_name());
- constraint.setDc_name(dcName);
- newConstraints.put(dcName, constraint);
- size += getObjectSize(SQLDefaultConstraint.class, constraint);
}
- defaultConstraintCache = newConstraints;
- updateMemberSize(MemberName.DEFAULT_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Default constraints refresh in cache was successful for {}.{}.{}", this.getTable().getCatName(),
- this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshCheckConstraints(List<SQLCheckConstraint> constraints) {
- Map<String, SQLCheckConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLCheckConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.CHECK_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping check constraint cache update for table: " + getTable().getTableName()
- + "; the check constraint are already refreshed.");
- return;
+ notNullConstraintCache = newNn;
+ break;
+ case DEFAULT_CONSTRAINT_CACHE:
+ Map<String, SQLDefaultConstraint> newDc = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLDefaultConstraint constraint : (List<SQLDefaultConstraint>) constraints) {
+ String dcName = StringUtils.normalizeIdentifier(constraint.getDc_name());
+ constraint.setDc_name(dcName);
+ newDc.put(dcName, constraint);
+ size += getObjectSize(SQLDefaultConstraint.class, constraint);
}
- String ccName = StringUtils.normalizeIdentifier(constraint.getDc_name());
- constraint.setDc_name(ccName);
- newConstraints.put(ccName, constraint);
- size += getObjectSize(SQLCheckConstraint.class, constraint);
}
- checkConstraintCache = newConstraints;
- updateMemberSize(MemberName.CHECK_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("check constraints refresh in cache was successful for {}.{}.{}", this.getTable().getCatName(),
- this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
+ defaultConstraintCache = newDc;
+ break;
+ case CHECK_CONSTRAINT_CACHE:
+ Map<String, SQLCheckConstraint> newCc = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLCheckConstraint constraint : (List<SQLCheckConstraint>) constraints) {
+ String ccName = StringUtils.normalizeIdentifier(constraint.getDc_name());
+ constraint.setDc_name(ccName);
+ newCc.put(ccName, constraint);
+ size += getObjectSize(SQLCheckConstraint.class, constraint);
+ }
+ }
+ checkConstraintCache = newCc;
+ break;
+ default:
+ LOG.error("Should not reach here");
+ break;
+ }
+ updateMemberSize(mn, size, SizeMode.Snapshot);
+ setMemberCacheUpdated(mn, false);
Review comment:
because it is a snapshot
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r578260034
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2674,28 +2554,19 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction()) || !sharedCache
Review comment:
This check is done in several places. Shall add utility method for this.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java
##########
@@ -29,11 +30,14 @@
private final Table table;
private final boolean isReplicated;
+ private final SQLAllTableConstraints constraints;
- public CreateTableEvent(Table table, boolean status, IHMSHandler handler, boolean isReplicated) {
+ public CreateTableEvent(Table table, boolean status, IHMSHandler handler, boolean isReplicated,
+ SQLAllTableConstraints constraints) {
Review comment:
We have separate events for add individual constraints. I think, we need not keep it as part of create table event.
It adds additional overhead when transmit events for incremental replication. Also, takes additional storage in HMS.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2187,8 +2160,10 @@ public void addPrimaryKeysToCache(String catName, String dbName, String tblName,
cacheLock.readLock().lock();
String tblKey = CacheUtils.buildTableKey(catName, dbName, tblName);
TableWrapper tblWrapper = tableCache.getIfPresent(tblKey);
- if (tblWrapper != null) {
- tblWrapper.cachePrimaryKeys(keys, false);
+ if ((tblWrapper != null) && tblWrapper.isConstraintsValid()) {
+ // Because lake of snapshot freshness validation.
+ // For now disabling cached constraint snapshot addition in parts by invalidating constraint snapshot.
+ tblWrapper.inValidateConstraints();
Review comment:
invalidate is single word in this context.Change method name to invalidateConstraintsCache.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r578572870
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2187,8 +2160,10 @@ public void addPrimaryKeysToCache(String catName, String dbName, String tblName,
cacheLock.readLock().lock();
String tblKey = CacheUtils.buildTableKey(catName, dbName, tblName);
TableWrapper tblWrapper = tableCache.getIfPresent(tblKey);
- if (tblWrapper != null) {
- tblWrapper.cachePrimaryKeys(keys, false);
+ if ((tblWrapper != null) && tblWrapper.isConstraintsValid()) {
+ // Because lake of snapshot freshness validation.
+ // For now disabling cached constraint snapshot addition in parts by invalidating constraint snapshot.
+ tblWrapper.inValidateConstraints();
Review comment:
Resolved
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2674,28 +2554,19 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction()) || !sharedCache
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r574409143
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -1788,47 +1771,16 @@ public boolean populateTableInCache(Table table, TableCacheObjects cacheObjects)
tblWrapper.setMemberCacheUpdated(MemberName.PARTITION_COL_STATS_CACHE, false);
tblWrapper.setMemberCacheUpdated(MemberName.AGGR_COL_STATS_CACHE, false);
- if (CollectionUtils.isNotEmpty(constraints.getPrimaryKeys())) {
- if (!tblWrapper.cachePrimaryKeys(constraints.getPrimaryKeys(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getForeignKeys())) {
- if (!tblWrapper.cacheForeignKeys(constraints.getForeignKeys(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.FOREIGN_KEY_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getNotNullConstraints())) {
- if (!tblWrapper.cacheNotNullConstraints(constraints.getNotNullConstraints(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.NOTNULL_CONSTRAINT_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getUniqueConstraints())) {
- if (!tblWrapper.cacheUniqueConstraints(constraints.getUniqueConstraints(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.UNIQUE_CONSTRAINT_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getDefaultConstraints())) {
- if (!tblWrapper.cacheDefaultConstraints(constraints.getDefaultConstraints(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getCheckConstraints())) {
- if (!tblWrapper.cacheCheckConstraints(constraints.getCheckConstraints(), true)) {
- return false;
- }
+ if(tblWrapper.cacheConstraints(constraints,true)) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r532042079
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -134,17 +134,8 @@ public int getPosition() {
}
}
- private enum MemberName {
- TABLE_COL_STATS_CACHE,
- PARTITION_CACHE,
- PARTITION_COL_STATS_CACHE,
- AGGR_COL_STATS_CACHE,
- PRIMARY_KEY_CACHE,
- FOREIGN_KEY_CACHE,
- NOTNULL_CONSTRAINT_CACHE,
- UNIQUE_CONSTRAINT_CACHE,
- DEFAULT_CONSTRAINT_CACHE,
- CHECK_CONSTRAINT_CACHE
+ public enum MemberName {
+ TABLE_COL_STATS_CACHE, PARTITION_CACHE, PARTITION_COL_STATS_CACHE, AGGR_COL_STATS_CACHE, PRIMARY_KEY_CACHE, FOREIGN_KEY_CACHE, NOTNULL_CONSTRAINT_CACHE, UNIQUE_CONSTRAINT_CACHE, DEFAULT_CONSTRAINT_CACHE, CHECK_CONSTRAINT_CACHE, ALL_TABLE_CONSTRAINT
Review comment:
Keep one member per line to be more readable.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -309,12 +302,19 @@ public int getObjectSize(Class<?> clazz, Object obj) {
private Map<String, SQLDefaultConstraint> defaultConstraintCache = new ConcurrentHashMap<>();
private Map<String, SQLCheckConstraint> checkConstraintCache = new ConcurrentHashMap<>();
+ private boolean pk = false;
Review comment:
Using too many flags is complex. Check if we can simplify it using below logic.
1) Fetch all constraints and update cache -> If this update is successful, then set "isAllConstraintsSet=true"
2) If partially successful due to memory constraint, then set "isAllConstraintsSet=false" and set bit position of completed constraint "long constraintStatusBits". We can assign one bit per constraintType and generally in the order in which we update it. It means, if we encounter a bit which is unset, then rest of the bits are unset too.
3) We never refresh individual constraints. Always try to update all constraints by using same method which follow above logic to set the flag. Shall remove all individual update(CachedStore.updateTablePrimaryKeys...) and refresh(SharedCache.refreshPrimaryKeysInCache...) methods for each types. This ensure, we always cache a valid snapshot.
4) When read any constraint, first check isAllConstraintsSet==true. If true, then assume all are valid and return the cached entry immediately. If false, then just the corresponding bit in "constraintStatusBits" and if true, then return the cached entry or else return null and invoke RawStore.
Note: We can also get rid of isAllConstraintsSet flag by using constraintStatusBits = -1 for "true" scenario.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get all constraint api
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r579648200
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##########
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern) {
}
return false;
}
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String dbName,
+ String tblName, boolean canUseEvents, RawStore rawStore) {
+ return !shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction());
+ }
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(String catName,
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r519879870
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2844,15 +2814,10 @@ public SQLAllTableConstraints getAllTableConstraints(String catName, String dbNa
return rawStore.getAllTableConstraints(catName, dbName, tblName);
}
- Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
- if (tbl == null) {
- // The table containing the constraints is not yet loaded in cache
- return rawStore.getAllTableConstraints(catName, dbName, tblName);
- }
SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
- // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
- // So fall back to raw store for correct values
+ /* If constraint value is missing then there might be the case that table is not stored in cached or
Review comment:
Add a TODO that this check is inefficient as all calls will hit rawstore if even one of the constraint type is not set for the given table.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2397,7 +2397,7 @@ public SQLAllTableConstraints listCachedAllTableConstraints(String catName, Stri
public List<SQLForeignKey> listCachedForeignKeys(String catName, String foreignDbName, String foreignTblName,
String parentDbName, String parentTblName) {
- List<SQLForeignKey> keys = new ArrayList<>();
+ List<SQLForeignKey> keys = null;
Review comment:
keys with null is accessed at line 2413
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Why is it hard to set a flag for consistency? If we hit else flow in this case, we shall try to update the cache and if that fails for some reason, then set the flag to false else set to true. Same with pre-warm and refresh flows.
Also, we need to check only the flag to validate the consistency of cache. No need to check for null or empty constraints.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r574995509
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2397,7 +2397,7 @@ public SQLAllTableConstraints listCachedAllTableConstraints(String catName, Stri
public List<SQLForeignKey> listCachedForeignKeys(String catName, String foreignDbName, String foreignTblName,
String parentDbName, String parentTblName) {
- List<SQLForeignKey> keys = new ArrayList<>();
+ List<SQLForeignKey> keys = null;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise getAlltableConstraint from 6 cache call to 1 cache call
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r511995335
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Majority of the calls are likely to be redirected to RawStore and lose the advantage of cache. How about an optional flag in SQLAllTableConstraints and TableWrapper to mark if it is a complete snapshot of constraints?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r570703754
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -309,12 +302,19 @@ public int getObjectSize(Class<?> clazz, Object obj) {
private Map<String, SQLDefaultConstraint> defaultConstraintCache = new ConcurrentHashMap<>();
private Map<String, SQLCheckConstraint> checkConstraintCache = new ConcurrentHashMap<>();
+ private boolean pk = false;
Review comment:
Changes are as follows
1. Replace 6 flag with 1 flag
2. In cache refresh/update flow removed 6 call to raw metastore with getAllTableConstraints()
3. While adding table to cache via createTable and createTableWithConstraint isConstraintValid flag is set as true.
4. If isConstraintValid flag is true then return data from cache else redirect to raw metastore.
5. While adding individual constraint to metastore if isConstraintValid is true that means constraints snapshot is valid and new constraint can be updated in cachestore else skip update.
6. isConstraintValid will only be true if all the constraint value are valid in the metastore else redirect to raw metastore (For the case of partial constraint storage due to memory shortage)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r531158744
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2844,15 +2814,10 @@ public SQLAllTableConstraints getAllTableConstraints(String catName, String dbNa
return rawStore.getAllTableConstraints(catName, dbName, tblName);
}
- Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
- if (tbl == null) {
- // The table containing the constraints is not yet loaded in cache
- return rawStore.getAllTableConstraints(catName, dbName, tblName);
- }
SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
- // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
- // So fall back to raw store for correct values
+ /* If constraint value is missing then there might be the case that table is not stored in cached or
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1610:
URL: https://github.com/apache/hive/pull/1610#issuecomment-768679497
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r514829244
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Adding a flag required bigger change. So for now I am reducing the scope of this PR to optimise following
1. Check only once if table exit in cached store.
2. Instead of calling individual constraint in cached store. Add a method which return all constraint at once and if data is not consistent then fall back to rawstore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get all constraint api
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r579648189
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##########
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern) {
}
return false;
}
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String dbName,
Review comment:
removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r574409517
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2187,7 +2139,7 @@ public void addPrimaryKeysToCache(String catName, String dbName, String tblName,
cacheLock.readLock().lock();
String tblKey = CacheUtils.buildTableKey(catName, dbName, tblName);
TableWrapper tblWrapper = tableCache.getIfPresent(tblKey);
- if (tblWrapper != null) {
+ if (tblWrapper != null && tblWrapper.isConstraintsValid()) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise getAlltableConstraint from 6 cache call to 1 cache call
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r512564828
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Also, what if the table just has primary keys and no other constraints?
nonEmpty(otherConstraints) will return true and we will not cache anything?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r574408965
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1325,6 +1204,7 @@ private void validateTableType(Table tbl) {
}
validateTableType(tbl);
sharedCache.addTableToCache(catName, dbName, tblName, tbl);
+ sharedCache.addTableConstraintsToCache(catName,dbName,tblName,new SQLAllTableConstraints());
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get all constraint api
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r579358426
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##########
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern) {
}
return false;
}
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String dbName,
Review comment:
It is not used anywhere other than below method. We can remove it and move these validations there.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##########
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern) {
}
return false;
}
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String dbName,
+ String tblName, boolean canUseEvents, RawStore rawStore) {
+ return !shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction());
+ }
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(String catName,
Review comment:
The method name is too long and not readable. Shall make it shouldGetConstraintFromRawStore().
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2554,8 +2554,7 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction()) || !sharedCache
- .isTableConstraintValid(catName, dbName, tblName)) {
+ if (validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(catName,dbName,tblName,canUseEvents,rawStore,sharedCache)) {
Review comment:
nit: Add space after each parameter.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##########
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern) {
}
return false;
}
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String dbName,
+ String tblName, boolean canUseEvents, RawStore rawStore) {
+ return !shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction());
+ }
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(String catName,
Review comment:
Shall move this method as private method inside CachedStore to avoid passing too many arguments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r531158652
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2397,7 +2397,7 @@ public SQLAllTableConstraints listCachedAllTableConstraints(String catName, Stri
public List<SQLForeignKey> listCachedForeignKeys(String catName, String foreignDbName, String foreignTblName,
String parentDbName, String parentTblName) {
- List<SQLForeignKey> keys = new ArrayList<>();
+ List<SQLForeignKey> keys = null;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get all constraint api
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r579648239
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CacheUtils.java
##########
@@ -144,4 +147,15 @@ public static boolean matches(String name, String pattern) {
}
return false;
}
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransaction(String catName, String dbName,
+ String tblName, boolean canUseEvents, RawStore rawStore) {
+ return !shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction());
+ }
+
+ public static boolean validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(String catName,
Review comment:
done
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2554,8 +2554,7 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction()) || !sharedCache
- .isTableConstraintValid(catName, dbName, tblName)) {
+ if (validateShouldCacheTableCanUseEventIsActiveTransactionIsConstraintsValid(catName,dbName,tblName,canUseEvents,rawStore,sharedCache)) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r575038955
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
##########
@@ -1558,6 +1558,7 @@ public void testPrimaryKeys() {
Table tbl = createUnpartitionedTableObject(db);
sharedCache.addTableToCache(DEFAULT_CATALOG_NAME, "db", tbl.getTableName(), tbl);
+ sharedCache.addTableConstraintsToCache(DEFAULT_CATALOG_NAME,"db", tbl.getTableName(), new SQLAllTableConstraints());
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r570703754
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -309,12 +302,19 @@ public int getObjectSize(Class<?> clazz, Object obj) {
private Map<String, SQLDefaultConstraint> defaultConstraintCache = new ConcurrentHashMap<>();
private Map<String, SQLCheckConstraint> checkConstraintCache = new ConcurrentHashMap<>();
+ private boolean pk = false;
Review comment:
Changes are as follows
1. Replace 6 flag with 1 flag
2. In cache refresh/update flow removed 6 call to raw metastore with getAllTableConstraints()
3. While adding table to cache via createTable and createTableWithConstraint isConstraintValid flag is set as true.
4. If isConstraintValid flag is true then return data from cache else redirect to raw metastore.
5. While adding individual constraint to metastore if isConstraintValid is true that means constraints snapshot is valid and new constraint can be updated in cachestore else skip update.
6. isConstraintValid will only be true if all the constraint value are valid in the metastore else redirect to raw metastore (For the case of partial constraint storage due to memory shortage)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r572168261
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1325,6 +1204,7 @@ private void validateTableType(Table tbl) {
}
validateTableType(tbl);
sharedCache.addTableToCache(catName, dbName, tblName, tbl);
+ sharedCache.addTableConstraintsToCache(catName,dbName,tblName,new SQLAllTableConstraints());
Review comment:
There is a separate api for createTableWithConstraints.
Is this api also used when table is created with constraints?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2814,21 +2649,12 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction()) || !sharedCache
+ .isTableConstraintValid(catName, dbName, tblName)) {
return rawStore.getCheckConstraints(catName, dbName, tblName);
}
+ return sharedCache.listCachedCheckConstraint(catName, dbName, tblName);
Review comment:
below check was present earlier:
```
if (CollectionUtils.isEmpty(keys)) {
return rawStore.getCheckConstraints(catName, dbName, tblName);
}
```
Why is this removed? Is this check moved somewhere else?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -418,12 +424,12 @@ private void updateMemberSize(MemberName mn, Integer size, SizeMode mode) {
}
switch (mode) {
- case Delta:
- this.memberObjectsSize[mn.ordinal()] += size;
- break;
- case Snapshot:
- this.memberObjectsSize[mn.ordinal()] = size;
- break;
+ case Delta:
Review comment:
nit: need extra spaces before case?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r575929152
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -715,154 +747,105 @@ public void removeConstraint(String name) {
}
}
- public void refreshPrimaryKeys(List<SQLPrimaryKey> keys) {
- Map<String, SQLPrimaryKey> newKeys = new ConcurrentHashMap<>();
+ public void refreshAllTableConstraints(SQLAllTableConstraints constraints) {
try {
tableLock.writeLock().lock();
- int size = 0;
- for (SQLPrimaryKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, true, false)) {
- LOG.debug("Skipping primary key cache update for table: " + getTable().getTableName()
- + "; the primary keys are already refreshed.");
- return;
- }
- String pkName = StringUtils.normalizeIdentifier(key.getPk_name());
- key.setPk_name(pkName);
- newKeys.put(pkName, key);
- size += getObjectSize(SQLPrimaryKey.class, key);
- }
- primaryKeyCache = newKeys;
- updateMemberSize(MemberName.PRIMARY_KEY_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Primary keys refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
+ this.isConstraintsValid =
+ refreshConstraint(constraints.getPrimaryKeys(), MemberName.PRIMARY_KEY_CACHE) && refreshConstraint(
+ constraints.getForeignKeys(), MemberName.FOREIGN_KEY_CACHE) && refreshConstraint(
+ constraints.getUniqueConstraints(), MemberName.UNIQUE_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getDefaultConstraints(), MemberName.DEFAULT_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getNotNullConstraints(), MemberName.NOTNULL_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getCheckConstraints(), MemberName.CHECK_CONSTRAINT_CACHE);
} finally {
tableLock.writeLock().unlock();
}
}
- public void refreshForeignKeys(List<SQLForeignKey> keys) {
- Map<String, SQLForeignKey> newKeys = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLForeignKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.FOREIGN_KEY_CACHE, true, false)) {
- LOG.debug("Skipping foreign key cache update for table: " + getTable().getTableName()
- + "; the foreign keys are already refreshed.");
- return;
+ private boolean refreshConstraint(List constraints, MemberName mn) {
Review comment:
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise getAlltableConstraint from 6 cache call to 1 cache call
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r512562376
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
+1.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Also, what if the table just a primary keys and not other constraints?
nonEmpty(otherConstraints) will return true and we will not cache anything?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r574298575
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1325,6 +1204,7 @@ private void validateTableType(Table tbl) {
}
validateTableType(tbl);
sharedCache.addTableToCache(catName, dbName, tblName, tbl);
+ sharedCache.addTableConstraintsToCache(catName,dbName,tblName,new SQLAllTableConstraints());
Review comment:
nit: Add space after comma.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -506,10 +512,18 @@ boolean cacheCheckConstraints(List<SQLCheckConstraint> checkConstraints, boolean
return cacheConstraints(checkConstraints, fromPrewarm, MemberName.CHECK_CONSTRAINT_CACHE);
}
+ boolean cacheConstraints(SQLAllTableConstraints constraints, boolean fromPrewarm) {
+ this.isConstraintsValid =
+ cacheConstraints(constraints.getPrimaryKeys(), fromPrewarm, MemberName.PRIMARY_KEY_CACHE) && cacheConstraints(
+ constraints.getForeignKeys(), fromPrewarm, MemberName.FOREIGN_KEY_CACHE) && cacheConstraints(constraints.getDefaultConstraints(), fromPrewarm, MemberName.DEFAULT_CONSTRAINT_CACHE)
+ && cacheConstraints(constraints.getUniqueConstraints(), fromPrewarm, MemberName.UNIQUE_CONSTRAINT_CACHE)
+ && cacheConstraints(constraints.getNotNullConstraints(), fromPrewarm, MemberName.NOTNULL_CONSTRAINT_CACHE)
+ && cacheConstraints(constraints.getCheckConstraints(), fromPrewarm, MemberName.CHECK_CONSTRAINT_CACHE);
+ return this.isConstraintsValid;
+ }
+
// Common method to cache constraints
- private boolean cacheConstraints(List constraintsList,
- boolean fromPrewarm,
- MemberName mn) {
+ private boolean cacheConstraints(List constraintsList, boolean fromPrewarm, MemberName mn) {
Review comment:
This method never returns false. It means, isConstraintsValid flag is always true. Is it by design? If yes, when it will be false?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2187,7 +2139,7 @@ public void addPrimaryKeysToCache(String catName, String dbName, String tblName,
cacheLock.readLock().lock();
String tblKey = CacheUtils.buildTableKey(catName, dbName, tblName);
TableWrapper tblWrapper = tableCache.getIfPresent(tblKey);
- if (tblWrapper != null) {
+ if (tblWrapper != null && tblWrapper.isConstraintsValid()) {
Review comment:
Keep tblWrapper != null within ().
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -1788,47 +1771,16 @@ public boolean populateTableInCache(Table table, TableCacheObjects cacheObjects)
tblWrapper.setMemberCacheUpdated(MemberName.PARTITION_COL_STATS_CACHE, false);
tblWrapper.setMemberCacheUpdated(MemberName.AGGR_COL_STATS_CACHE, false);
- if (CollectionUtils.isNotEmpty(constraints.getPrimaryKeys())) {
- if (!tblWrapper.cachePrimaryKeys(constraints.getPrimaryKeys(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getForeignKeys())) {
- if (!tblWrapper.cacheForeignKeys(constraints.getForeignKeys(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.FOREIGN_KEY_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getNotNullConstraints())) {
- if (!tblWrapper.cacheNotNullConstraints(constraints.getNotNullConstraints(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.NOTNULL_CONSTRAINT_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getUniqueConstraints())) {
- if (!tblWrapper.cacheUniqueConstraints(constraints.getUniqueConstraints(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.UNIQUE_CONSTRAINT_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getDefaultConstraints())) {
- if (!tblWrapper.cacheDefaultConstraints(constraints.getDefaultConstraints(), true)) {
- return false;
- }
- }
- tblWrapper.setMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, false);
-
- if (CollectionUtils.isNotEmpty(constraints.getCheckConstraints())) {
- if (!tblWrapper.cacheCheckConstraints(constraints.getCheckConstraints(), true)) {
- return false;
- }
+ if(tblWrapper.cacheConstraints(constraints,true)) {
Review comment:
nit: Space before ( and after comma
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
##########
@@ -1558,6 +1558,7 @@ public void testPrimaryKeys() {
Table tbl = createUnpartitionedTableObject(db);
sharedCache.addTableToCache(DEFAULT_CATALOG_NAME, "db", tbl.getTableName(), tbl);
+ sharedCache.addTableConstraintsToCache(DEFAULT_CATALOG_NAME,"db", tbl.getTableName(), new SQLAllTableConstraints());
Review comment:
nit: Need space after comma
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2397,7 +2397,7 @@ public SQLAllTableConstraints listCachedAllTableConstraints(String catName, Stri
public List<SQLForeignKey> listCachedForeignKeys(String catName, String foreignDbName, String foreignTblName,
String parentDbName, String parentTblName) {
- List<SQLForeignKey> keys = new ArrayList<>();
+ List<SQLForeignKey> keys = null;
Review comment:
Still keys can be referenced with null at line 2365.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -715,154 +747,105 @@ public void removeConstraint(String name) {
}
}
- public void refreshPrimaryKeys(List<SQLPrimaryKey> keys) {
- Map<String, SQLPrimaryKey> newKeys = new ConcurrentHashMap<>();
+ public void refreshAllTableConstraints(SQLAllTableConstraints constraints) {
try {
tableLock.writeLock().lock();
- int size = 0;
- for (SQLPrimaryKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, true, false)) {
- LOG.debug("Skipping primary key cache update for table: " + getTable().getTableName()
- + "; the primary keys are already refreshed.");
- return;
- }
- String pkName = StringUtils.normalizeIdentifier(key.getPk_name());
- key.setPk_name(pkName);
- newKeys.put(pkName, key);
- size += getObjectSize(SQLPrimaryKey.class, key);
- }
- primaryKeyCache = newKeys;
- updateMemberSize(MemberName.PRIMARY_KEY_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Primary keys refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
+ this.isConstraintsValid =
+ refreshConstraint(constraints.getPrimaryKeys(), MemberName.PRIMARY_KEY_CACHE) && refreshConstraint(
+ constraints.getForeignKeys(), MemberName.FOREIGN_KEY_CACHE) && refreshConstraint(
+ constraints.getUniqueConstraints(), MemberName.UNIQUE_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getDefaultConstraints(), MemberName.DEFAULT_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getNotNullConstraints(), MemberName.NOTNULL_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getCheckConstraints(), MemberName.CHECK_CONSTRAINT_CACHE);
} finally {
tableLock.writeLock().unlock();
}
}
- public void refreshForeignKeys(List<SQLForeignKey> keys) {
- Map<String, SQLForeignKey> newKeys = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLForeignKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.FOREIGN_KEY_CACHE, true, false)) {
- LOG.debug("Skipping foreign key cache update for table: " + getTable().getTableName()
- + "; the foreign keys are already refreshed.");
- return;
+ private boolean refreshConstraint(List constraints, MemberName mn) {
+ int size = 0;
+ switch (mn) {
+ case PRIMARY_KEY_CACHE:
+ Map<String, SQLPrimaryKey> newPk = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLPrimaryKey key : (List<SQLPrimaryKey>) constraints) {
+ String pkName = StringUtils.normalizeIdentifier(key.getPk_name());
+ key.setPk_name(pkName);
+ newPk.put(pkName, key);
+ size += getObjectSize(SQLPrimaryKey.class, key);
}
- String fkName = StringUtils.normalizeIdentifier(key.getFk_name());
- key.setFk_name(fkName);
- newKeys.put(fkName, key);
- size += getObjectSize(SQLForeignKey.class, key);
}
- foreignKeyCache = newKeys;
- updateMemberSize(MemberName.FOREIGN_KEY_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Foreign keys refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshNotNullConstraints(List<SQLNotNullConstraint> constraints) {
- Map<String, SQLNotNullConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLNotNullConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.NOTNULL_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping not null constraints cache update for table: " + getTable().getTableName()
- + "; the not null constraints are already refreshed.");
- return;
+ primaryKeyCache = newPk;
+ break;
+ case FOREIGN_KEY_CACHE:
+ Map<String, SQLForeignKey> newFk = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLForeignKey key : (List<SQLForeignKey>) constraints) {
+ String fkName = StringUtils.normalizeIdentifier(key.getFk_name());
+ key.setFk_name(fkName);
+ newFk.put(fkName, key);
+ size += getObjectSize(SQLForeignKey.class, key);
}
- String nnName = StringUtils.normalizeIdentifier(constraint.getNn_name());
- constraint.setNn_name(nnName);
- newConstraints.put(nnName, constraint);
- size += getObjectSize(SQLNotNullConstraint.class, constraint);
}
- notNullConstraintCache = newConstraints;
- updateMemberSize(MemberName.NOTNULL_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Not null constraints refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshUniqueConstraints(List<SQLUniqueConstraint> constraints) {
- Map<String, SQLUniqueConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLUniqueConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.UNIQUE_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping unique constraints cache update for table: " + getTable().getTableName()
- + "; the unique constraints are already refreshed.");
- return;
+ foreignKeyCache = newFk;
+ break;
+ case UNIQUE_CONSTRAINT_CACHE:
+ Map<String, SQLUniqueConstraint> newUc = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLUniqueConstraint constraint : (List<SQLUniqueConstraint>) constraints) {
+ String ucName = StringUtils.normalizeIdentifier(constraint.getUk_name());
+ constraint.setUk_name(ucName);
+ newUc.put(ucName, constraint);
+ size += getObjectSize(SQLUniqueConstraint.class, constraint);
}
- String ucName = StringUtils.normalizeIdentifier(constraint.getUk_name());
- constraint.setUk_name(ucName);
- newConstraints.put(ucName, constraint);
- size += getObjectSize(SQLUniqueConstraint.class, constraint);
}
- uniqueConstraintCache = newConstraints;
- updateMemberSize(MemberName.UNIQUE_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Unique constraints refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshDefaultConstraints(List<SQLDefaultConstraint> constraints) {
- Map<String, SQLDefaultConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLDefaultConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping default constraint cache update for table: " + getTable().getTableName()
- + "; the default constraint are already refreshed.");
- return;
+ uniqueConstraintCache = newUc;
+ break;
+ case NOTNULL_CONSTRAINT_CACHE:
+ Map<String, SQLNotNullConstraint> newNn = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLNotNullConstraint constraint : (List<SQLNotNullConstraint>) constraints) {
+ String nnName = StringUtils.normalizeIdentifier(constraint.getNn_name());
+ constraint.setNn_name(nnName);
+ newNn.put(nnName, constraint);
+ size += getObjectSize(SQLNotNullConstraint.class, constraint);
}
- String dcName = StringUtils.normalizeIdentifier(constraint.getDc_name());
- constraint.setDc_name(dcName);
- newConstraints.put(dcName, constraint);
- size += getObjectSize(SQLDefaultConstraint.class, constraint);
}
- defaultConstraintCache = newConstraints;
- updateMemberSize(MemberName.DEFAULT_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Default constraints refresh in cache was successful for {}.{}.{}", this.getTable().getCatName(),
- this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
- }
-
- public void refreshCheckConstraints(List<SQLCheckConstraint> constraints) {
- Map<String, SQLCheckConstraint> newConstraints = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLCheckConstraint constraint : constraints) {
- if (compareAndSetMemberCacheUpdated(MemberName.CHECK_CONSTRAINT_CACHE, true, false)) {
- LOG.debug("Skipping check constraint cache update for table: " + getTable().getTableName()
- + "; the check constraint are already refreshed.");
- return;
+ notNullConstraintCache = newNn;
+ break;
+ case DEFAULT_CONSTRAINT_CACHE:
+ Map<String, SQLDefaultConstraint> newDc = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLDefaultConstraint constraint : (List<SQLDefaultConstraint>) constraints) {
+ String dcName = StringUtils.normalizeIdentifier(constraint.getDc_name());
+ constraint.setDc_name(dcName);
+ newDc.put(dcName, constraint);
+ size += getObjectSize(SQLDefaultConstraint.class, constraint);
}
- String ccName = StringUtils.normalizeIdentifier(constraint.getDc_name());
- constraint.setDc_name(ccName);
- newConstraints.put(ccName, constraint);
- size += getObjectSize(SQLCheckConstraint.class, constraint);
}
- checkConstraintCache = newConstraints;
- updateMemberSize(MemberName.CHECK_CONSTRAINT_CACHE, size, SizeMode.Snapshot);
- LOG.debug("check constraints refresh in cache was successful for {}.{}.{}", this.getTable().getCatName(),
- this.getTable().getDbName(), this.getTable().getTableName());
- } finally {
- tableLock.writeLock().unlock();
- }
+ defaultConstraintCache = newDc;
+ break;
+ case CHECK_CONSTRAINT_CACHE:
+ Map<String, SQLCheckConstraint> newCc = new ConcurrentHashMap<>();
+ if (!CollectionUtils.isEmpty(constraints)) {
+ for (SQLCheckConstraint constraint : (List<SQLCheckConstraint>) constraints) {
+ String ccName = StringUtils.normalizeIdentifier(constraint.getDc_name());
+ constraint.setDc_name(ccName);
+ newCc.put(ccName, constraint);
+ size += getObjectSize(SQLCheckConstraint.class, constraint);
+ }
+ }
+ checkConstraintCache = newCc;
+ break;
+ default:
+ LOG.error("Should not reach here");
+ break;
+ }
+ updateMemberSize(mn, size, SizeMode.Snapshot);
+ setMemberCacheUpdated(mn, false);
Review comment:
Why do we set this flag to false?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2187,7 +2139,7 @@ public void addPrimaryKeysToCache(String catName, String dbName, String tblName,
cacheLock.readLock().lock();
String tblKey = CacheUtils.buildTableKey(catName, dbName, tblName);
TableWrapper tblWrapper = tableCache.getIfPresent(tblKey);
- if (tblWrapper != null) {
+ if (tblWrapper != null && tblWrapper.isConstraintsValid()) {
Review comment:
Why do we update cache for each constraint individually? We should always cache a single snapshot of constraints of table. If not, isConstraintsValid flag wouldn't make sense.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -715,154 +747,105 @@ public void removeConstraint(String name) {
}
}
- public void refreshPrimaryKeys(List<SQLPrimaryKey> keys) {
- Map<String, SQLPrimaryKey> newKeys = new ConcurrentHashMap<>();
+ public void refreshAllTableConstraints(SQLAllTableConstraints constraints) {
try {
tableLock.writeLock().lock();
- int size = 0;
- for (SQLPrimaryKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, true, false)) {
- LOG.debug("Skipping primary key cache update for table: " + getTable().getTableName()
- + "; the primary keys are already refreshed.");
- return;
- }
- String pkName = StringUtils.normalizeIdentifier(key.getPk_name());
- key.setPk_name(pkName);
- newKeys.put(pkName, key);
- size += getObjectSize(SQLPrimaryKey.class, key);
- }
- primaryKeyCache = newKeys;
- updateMemberSize(MemberName.PRIMARY_KEY_CACHE, size, SizeMode.Snapshot);
- LOG.debug("Primary keys refresh in cache was successful for {}.{}.{}",
- this.getTable().getCatName(), this.getTable().getDbName(), this.getTable().getTableName());
+ this.isConstraintsValid =
+ refreshConstraint(constraints.getPrimaryKeys(), MemberName.PRIMARY_KEY_CACHE) && refreshConstraint(
+ constraints.getForeignKeys(), MemberName.FOREIGN_KEY_CACHE) && refreshConstraint(
+ constraints.getUniqueConstraints(), MemberName.UNIQUE_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getDefaultConstraints(), MemberName.DEFAULT_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getNotNullConstraints(), MemberName.NOTNULL_CONSTRAINT_CACHE) && refreshConstraint(
+ constraints.getCheckConstraints(), MemberName.CHECK_CONSTRAINT_CACHE);
} finally {
tableLock.writeLock().unlock();
}
}
- public void refreshForeignKeys(List<SQLForeignKey> keys) {
- Map<String, SQLForeignKey> newKeys = new ConcurrentHashMap<>();
- try {
- tableLock.writeLock().lock();
- int size = 0;
- for (SQLForeignKey key : keys) {
- if (compareAndSetMemberCacheUpdated(MemberName.FOREIGN_KEY_CACHE, true, false)) {
- LOG.debug("Skipping foreign key cache update for table: " + getTable().getTableName()
- + "; the foreign keys are already refreshed.");
- return;
+ private boolean refreshConstraint(List constraints, MemberName mn) {
Review comment:
This also always returns true and in tune make isConstraintsValid=true always.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r574408036
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2187,7 +2139,7 @@ public void addPrimaryKeysToCache(String catName, String dbName, String tblName,
cacheLock.readLock().lock();
String tblKey = CacheUtils.buildTableKey(catName, dbName, tblName);
TableWrapper tblWrapper = tableCache.getIfPresent(tblKey);
- if (tblWrapper != null) {
+ if (tblWrapper != null && tblWrapper.isConstraintsValid()) {
Review comment:
Q Why do we update cache for each constraint individually?
Ans - To update the cache for specific constraint for query like "ALTER TABLE table_name ADD CONSTRAINT" instead of invalidating the whole cache and updating again because that will lead to cache miss or extra network call is required to delete and update all cache constraints.
Q We should always cache a single snapshot of constraints of table. If not, isConstraintsValid flag wouldn't make sense.
Ans - So if isConstraintsValid is valid that means there is valid copy of constraint present then we can add the incoming constraint and save redundant call. if isConstraintsValid is false then skip because cached constraint are inaccessible and waiting for refreshed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r578567748
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/events/CreateTableEvent.java
##########
@@ -29,11 +30,14 @@
private final Table table;
private final boolean isReplicated;
+ private final SQLAllTableConstraints constraints;
- public CreateTableEvent(Table table, boolean status, IHMSHandler handler, boolean isReplicated) {
+ public CreateTableEvent(Table table, boolean status, IHMSHandler handler, boolean isReplicated,
+ SQLAllTableConstraints constraints) {
Review comment:
Removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh merged pull request #1610: HIVE-24259: [CachedStore] Optimise get all constraint api
Posted by GitBox <gi...@apache.org>.
sankarh merged pull request #1610:
URL: https://github.com/apache/hive/pull/1610
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise getAlltableConstraint from 6 cache call to 1 cache call
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r512565056
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Good idea. I will check how we can implement a flag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r531158438
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2836,14 +2836,32 @@ long getPartsFound() {
@Override
public SQLAllTableConstraints getAllTableConstraints(String catName, String dbName, String tblName)
throws MetaException, NoSuchObjectException {
- SQLAllTableConstraints sqlAllTableConstraints = new SQLAllTableConstraints();
- sqlAllTableConstraints.setPrimaryKeys(getPrimaryKeys(catName, dbName, tblName));
- sqlAllTableConstraints.setForeignKeys(getForeignKeys(catName, null, null, dbName, tblName));
- sqlAllTableConstraints.setUniqueConstraints(getUniqueConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setDefaultConstraints(getDefaultConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setCheckConstraints(getCheckConstraints(catName, dbName, tblName));
- sqlAllTableConstraints.setNotNullConstraints(getNotNullConstraints(catName, dbName, tblName));
- return sqlAllTableConstraints;
+
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+
+ Table tbl = sharedCache.getTableFromCache(catName, dbName, tblName);
+ if (tbl == null) {
+ // The table containing the constraints is not yet loaded in cache
+ return rawStore.getAllTableConstraints(catName, dbName, tblName);
+ }
+ SQLAllTableConstraints constraints = sharedCache.listCachedAllTableConstraints(catName, dbName, tblName);
+
+ // if any of the constraint value is missing then there might be the case of partial constraints are stored in cached.
+ // So fall back to raw store for correct values
+ if (constraints != null && CollectionUtils.isNotEmpty(constraints.getPrimaryKeys()) && CollectionUtils
Review comment:
Added flag
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r572723842
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1325,6 +1204,7 @@ private void validateTableType(Table tbl) {
}
validateTableType(tbl);
sharedCache.addTableToCache(catName, dbName, tblName, tbl);
+ sharedCache.addTableConstraintsToCache(catName,dbName,tblName,new SQLAllTableConstraints());
Review comment:
This line was added to update table cache with all constraints null and flag set to true. So avoid redirection to raw store.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #1610: HIVE-24259: [CachedStore] Optimise get constraints call by removing redundant table check
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #1610:
URL: https://github.com/apache/hive/pull/1610#discussion_r572726538
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -2814,21 +2649,12 @@ long getPartsFound() {
catName = StringUtils.normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction()) || !sharedCache
+ .isTableConstraintValid(catName, dbName, tblName)) {
return rawStore.getCheckConstraints(catName, dbName, tblName);
}
+ return sharedCache.listCachedCheckConstraint(catName, dbName, tblName);
Review comment:
Added flag - isConstraintValid to check null value is consistent instead of going to raw store.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org