You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/03/28 09:15:18 UTC
[GitHub] [hive] ashish-kumar-sharma opened a new pull request #2128: [WIP]HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
ashish-kumar-sharma opened a new pull request #2128:
URL: https://github.com/apache/hive/pull/2128
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-839420970
@kishendas Could you please review this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642060827
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
Need to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-850964411
@sankarh This PR is about adding validWriteIdList as part of cached store.
Current status
1. HIVE-23931 - Send ValidWriteIdList and tableId to get_*_constraints HMS APIs (open)
2. HIVE-23820: [HS2] Send tableId in request for get_table_request API (merged)
3. HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper (open)
4. HIVE-23821: Send tableId in request for all the new HMS get_partition APIs (merged)
Once above PR get merged. I will raise a PR to add tableId and writedId check to all the methods and there corresponding test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642041067
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName, String tblName, String vali
catName = normalizeIdentifier(catName);
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] closed pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #2128:
URL: https://github.com/apache/hive/pull/2128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r636821684
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
A curious question.
I think, we recently added writeId allocation logic for DDL operations. Do we rollback the changes happened in metastore if DDL op fails?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
Probably, getting the snapshot before fetching the metadata objects might help to handle some cases.
1. Take snapshot from current thread and cache it -> ValidWriteIdList=:HWM=10:
2. Fetch table object --> writeid=10
2. Another thread writes table object --> writeid=11
3. In this case, new snapshot would be ValidWriteIdList=:HWM=11: which doesn't match with cached snapshot
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName, String tblName, String vali
catName = normalizeIdentifier(catName);
Review comment:
@Override
public Table getTable(String catName, String dbName, String tblName, String validWriteIds) throws MetaException {
return getTable(catName, dbName, tblName, null, -1);
}
Here, validWriteIds is not passed down.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
+ new GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName, tblName))))
+ .getTblValidWriteIds().get(0));
+ Deadline.stopTimer();
+ } catch (Exception e) {
+ LOG.info("Updating CachedStore: unable to update table validWriteIdList of catalog: " + catName + ", database: "
+ + dbName + ", table: " + tblName, e);
+ }
+ if (validWriteIdList != null) {
+ sharedCache.refreshValidWriteIdListInCache(catName, dbName, tblName, validWriteIdList);
+ LOG.debug("CachedStore: updated cached table validWriteIdList for catalog: {}, database: {}, table: {}",
Review comment:
Also, log the ValidWriteIdList.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2498,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isStale = true;
+
+ if (StringUtils.isEmpty(validWriteIdList))
+ return isStale;
Review comment:
It sounds like, if we cannot get ValidWriteIdList, then the cache is always marked as invalid. Can we not proceed to populate the table metadata into cache if fetching ValidWriteIdList fail for any reason?
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
ValidWriteIdList obtained here doesn't guarantee validity of cached metadata objects.
Consider below sequence.
1. Fetch table object --> writeid=10
2. Another thread writes table object --> writeid=11
3. Take snapshot from current thread and cache it -> ValidWriteIdList=<tbl>:HWM=11:
4. Now, the table object is stale but we still mark it as valid for any new txn.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName, String tblName, String vali
catName = normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ if (shouldGetTableFromRawStore(catName, dbName, tblName, validWriteIds)) {
Review comment:
We should use "shouldGetTableFromRawStore" even for get partitions, column stats.
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -573,6 +577,18 @@ static void prewarm(RawStore rawStore) {
Deadline.stopTimer();
cacheObjects.setTableConstraints(tableConstraints);
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ ValidWriteIdList validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore
+ .getValidWriteIds(
+ new GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName, tblName))))
+ .getTblValidWriteIds().get(0));
+ Deadline.stopTimer();
+ cacheObjects.setValidWriteIdList(validWriteIdList);
+ } catch (Exception e) {
+ LOG.debug(ExceptionUtils.getStackTrace(e));
Review comment:
It can be atleast an info log.
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
##########
@@ -38,10 +38,11 @@
import org.apache.hadoop.hive.metastore.Warehouse;
import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest;
import org.apache.hadoop.hive.metastore.api.*;
-import org.apache.hadoop.hive.metastore.api.utils.DecimalUtils;
import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder;
import org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector;
import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
Review comment:
Add tests to check if invalidation happens only for stale data or not. Also, after refresh it should be valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r608511275
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isValid = false;
+
+ if (StringUtils.isEmpty(validWriteIdList))
+ return false;
Review comment:
nit: add braces.
Also, should we return true? if validWriteId is not present, we can safely assume that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642041015
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -573,6 +577,18 @@ static void prewarm(RawStore rawStore) {
Deadline.stopTimer();
cacheObjects.setTableConstraints(tableConstraints);
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ ValidWriteIdList validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore
+ .getValidWriteIds(
+ new GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName, tblName))))
+ .getTblValidWriteIds().get(0));
+ Deadline.stopTimer();
+ cacheObjects.setValidWriteIdList(validWriteIdList);
+ } catch (Exception e) {
+ LOG.debug(ExceptionUtils.getStackTrace(e));
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-821000521
@kishendas Can you please review this PR mean while I will add UT in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-899899028
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642039659
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName, String tblName, String vali
catName = normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
- if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents && rawStore.isActiveTransaction())) {
+ if (shouldGetTableFromRawStore(catName, dbName, tblName, validWriteIds)) {
Review comment:
I will do that in new PR. Where I will put tableId and writeId check on all the methods in cacheStore. This PR is mostly around adding validWriteIdList to TableWrapper Object in cacheStore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642039720
##########
File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
##########
@@ -38,10 +38,11 @@
import org.apache.hadoop.hive.metastore.Warehouse;
import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest;
import org.apache.hadoop.hive.metastore.api.*;
-import org.apache.hadoop.hive.metastore.api.utils.DecimalUtils;
import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder;
import org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector;
import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
Review comment:
I will do that in new PR. Where I will put tableId and writeId check on all the methods and there corresponding tests in cacheStore. This PR is mostly around adding validWriteIdList to TableWrapper Object in cacheStore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642040720
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2498,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isStale = true;
+
+ if (StringUtils.isEmpty(validWriteIdList))
+ return isStale;
Review comment:
There is background updater thread which is running in certain interval. Which update all table object. We are targeting eventual consistency for cached store. https://cwiki.apache.org/confluence/display/Hive/Synchronized+Metastore+Cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660
@kgyrtkirk Could please review the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r648497223
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
AFAIK, we are targeting strict consistency with "Synchronized metastore cache" work. Eventual consistency is supported even today with regular refresh of cache with the interval of 1 min.
The problem is, it returns from cache even if it is invalid which is not right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660
@kgyrtkirk Could please review the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642060638
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
Since we are target eventual consistency it is fine we fall-back to raw store. But we can target aggressive refresh rate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r648497223
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
AFAIK, we are targeting strict consistency with "Synchronized metastore cache" work. Eventual consistency is supported even today with regular refresh of cache with the interval of 1 min. Pls recheck this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma edited a comment on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma edited a comment on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660
@kgyrtkirk Could you please review the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r614644380
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isValid = false;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sankarh commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r648501960
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2498,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isStale = true;
+
+ if (StringUtils.isEmpty(validWriteIdList))
+ return isStale;
Review comment:
Background thread is the current logic to refresh the cache at regular intervals. Synchronized cache work shouldn't rely on that as it should provide strict consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642041716
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##########
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore, String catName, String
}
}
+ private void updateValidWriteIdList(TxnStore txnStore, String catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+ Deadline.startTimer("getValidWriteIds");
+ validWriteIdList = TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
+ new GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName, tblName))))
+ .getTblValidWriteIds().get(0));
+ Deadline.stopTimer();
+ } catch (Exception e) {
+ LOG.info("Updating CachedStore: unable to update table validWriteIdList of catalog: " + catName + ", database: "
+ + dbName + ", table: " + tblName, e);
+ }
+ if (validWriteIdList != null) {
+ sharedCache.refreshValidWriteIdListInCache(catName, dbName, tblName, validWriteIdList);
+ LOG.debug("CachedStore: updated cached table validWriteIdList for catalog: {}, database: {}, table: {}",
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma edited a comment on pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma edited a comment on pull request #2128:
URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660
@kgyrtkirk Could you please review the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] ashish-kumar-sharma commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
ashish-kumar-sharma commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r614644293
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isValid = false;
+
+ if (StringUtils.isEmpty(validWriteIdList))
+ return false;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r608870539
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isValid = false;
Review comment:
nit: rename the variable to `isStale` instead? Below we are setting isValid to true when the the writeIdList in CachedStore is stale/older than client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] adesh-rao commented on a change in pull request #2128: HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper
Posted by GitBox <gi...@apache.org>.
adesh-rao commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r608511275
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##########
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName, String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String tblName, String validWriteIdList) {
+ boolean isValid = false;
+
+ if (StringUtils.isEmpty(validWriteIdList))
+ return false;
Review comment:
nit: add braces.
Also, should we return true? if validWriteId is not present, we can assume that table is not loaded properly and it should be fetched from metastore db.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org