You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "dengzhhu653 (via GitHub)" <gi...@apache.org> on 2025/11/14 12:35:23 UTC
[PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
dengzhhu653 opened a new pull request, #6188:
URL: https://github.com/apache/hive/pull/6188
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3539706126
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=ACCEPTED)
Measures
 [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547736050
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -256,7 +256,9 @@ public String toString() {
ConfVars.PARTITION_ORDER_EXPR,
ConfVars.CAPABILITY_CHECK,
ConfVars.DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES,
- ConfVars.EXPRESSION_PROXY_CLASS
+ ConfVars.EXPRESSION_PROXY_CLASS,
+ ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES,
+ ConfVars.METASTORE_S4U_NOWAIT_RETRY_SLEEP_INTERVAL
Review Comment:
good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2548194832
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -256,7 +256,9 @@ public String toString() {
ConfVars.PARTITION_ORDER_EXPR,
ConfVars.CAPABILITY_CHECK,
ConfVars.DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES,
- ConfVars.EXPRESSION_PROXY_CLASS
+ ConfVars.EXPRESSION_PROXY_CLASS,
+ ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES,
+ ConfVars.METASTORE_S4U_NOWAIT_RETRY_SLEEP_INTERVAL
Review Comment:
make sense, before that I was thinking for the client re-connecting when the two properties change.
but we can make it simpler, only `set metaconf:` will take effect, as these are Metastore only properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547736050
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -256,7 +256,9 @@ public String toString() {
ConfVars.PARTITION_ORDER_EXPR,
ConfVars.CAPABILITY_CHECK,
ConfVars.DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES,
- ConfVars.EXPRESSION_PROXY_CLASS
+ ConfVars.EXPRESSION_PROXY_CLASS,
+ ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES,
+ ConfVars.METASTORE_S4U_NOWAIT_RETRY_SLEEP_INTERVAL
Review Comment:
good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546590027
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3608,7 +3609,7 @@ protected Integer getJdoResult(GetHelper<Integer> ctx)
* you want results for. E.g., if resultsCol is partitionName, the Collection
* has types of String, and if resultsCol is null, the types are MPartition.
*/
- private Collection<String> getPartitionPsQueryResults(String catName, String dbName,
+ private <T> Collection<T> getPartitionPsQueryResults(String catName, String dbName,
Review Comment:
in listPartitionNamesPs, it returns the List<String>, a list of partition names
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546296468
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java:
##########
@@ -1102,6 +1102,7 @@ Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats, Strin
* @throws InvalidObjectException the stats object is invalid
* @throws InvalidInputException unable to record the stats for the table
*/
+ @Deprecated
Review Comment:
could we add clarification what should be used instead
+ https://sonarcloud.io/project/issues?sinceLeakPeriod=true&issueStatuses=OPEN%2CCONFIRMED&pullRequest=6188&id=apache_hive&open=AZqCpJfSFL9TKjVAgeFB
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java:
##########
@@ -1102,6 +1102,7 @@ Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats, Strin
* @throws InvalidObjectException the stats object is invalid
* @throws InvalidInputException unable to record the stats for the table
*/
+ @Deprecated
Review Comment:
could we add clarification what should be used instead
\+ https://sonarcloud.io/project/issues?sinceLeakPeriod=true&issueStatuses=OPEN%2CCONFIRMED&pullRequest=6188&id=apache_hive&open=AZqCpJfSFL9TKjVAgeFB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546590027
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3608,7 +3609,7 @@ protected Integer getJdoResult(GetHelper<Integer> ctx)
* you want results for. E.g., if resultsCol is partitionName, the Collection
* has types of String, and if resultsCol is null, the types are MPartition.
*/
- private Collection<String> getPartitionPsQueryResults(String catName, String dbName,
+ private <T> Collection<T> getPartitionPsQueryResults(String catName, String dbName,
Review Comment:
in listPartitionNamesPs, it returns the `List<String>`, a list of partition names
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547731956
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
Review Comment:
is it a client config or HMS server? can the user modify it from beeline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3539610294
> I would suggest to revert the original PR and keep the fix in 1 commit
I'm trying to keep the changes to be reviewed simple, and avoid a next cycle of reviews on the old one.
This part changes the identity-type of MPartition in the package.jdo, when porting it requires a bit consideration:
- If have the direct sql add partition(HIVE-26035), then need the HIVE-29030 ported as well.
- Otherwise it's safety to port.
The updateTableColumnStatistics doesn't have such a pre-requisite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547116191
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -256,7 +256,9 @@ public String toString() {
ConfVars.PARTITION_ORDER_EXPR,
ConfVars.CAPABILITY_CHECK,
ConfVars.DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES,
- ConfVars.EXPRESSION_PROXY_CLASS
+ ConfVars.EXPRESSION_PROXY_CLASS,
+ ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES,
+ ConfVars.METASTORE_S4U_NOWAIT_RETRY_SLEEP_INTERVAL
Review Comment:
I don't think we need this. L275-277 is good enough right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547731956
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
Review Comment:
i see you've added them to `User configurable Metastore vars`. How can the user modify it from beeline? is there an hql for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2527959496
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9246,93 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
+ Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(),
+ mPartition, TxnUtils.isAcidTable(table));
Review Comment:
The `partition` object is created outside the retry loop (lines 9262-9263) but used inside the retry loop (line 9298). After `pm.refresh(mPartition)` on line 9287, the `partition` object may contain stale data since it was converted from the old `mPartition` state before the refresh.
Consider converting the `Partition` object inside the retry loop after refreshing `mPartition`, similar to how the table column statistics update converts the `Table` object inside the retry loop (see line 9188).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546345814
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
+ long sleepInterval = MetastoreConf.getTimeVar(conf,
+ ConfVars.METASTORE_S4U_NOWAIT_RETRY_SLEEP_INTERVAL, TimeUnit.MILLISECONDS);
+ Map<String, String> result = new RetryingExecutor<>(maxRetries, () -> {
+ Ref<Exception> exceptionRef = new Ref<>();
+ String savePoint = "ups_" + ThreadLocalRandom.current().nextInt(10000) + "_" + System.nanoTime();
+ setTransactionSavePoint(savePoint);
+ executePlainSQL(sqlGenerator.addForUpdateNoWait(
+ "SELECT \"PART_ID\" FROM \"PARTITIONS\" WHERE \"PART_ID\" = " + mPartition.getId()),
+ true,
+ exception -> {
+ rollbackTransactionToSavePoint(savePoint);
+ exceptionRef.t = exception;
+ });
+ if (exceptionRef.t != null) {
+ throw new RetryingExecutor.RetryException(exceptionRef.t);
+ }
+ pm.refresh(mPartition);
+ Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(),
+ mPartition, TxnUtils.isAcidTable(table));
+ Map<String, MPartitionColumnStatistics> oldStats = Maps.newHashMap();
+ List<MPartitionColumnStatistics> stats =
+ getMPartitionColumnStatistics(table, Lists.newArrayList(statsDesc.getPartName()), colNames, colStats.getEngine());
+ for (MPartitionColumnStatistics cStat : stats) {
+ oldStats.put(cStat.getColName(), cStat);
+ }
+
+ for (ColumnStatisticsObj statsObj : statsObjs) {
+ MPartitionColumnStatistics mStatsObj = StatObjectConverter.convertToMPartitionColumnStatistics(mPartition,
+ statsDesc, statsObj, colStats.getEngine());
+ writeMPartitionColumnStatistics(table, partition, mStatsObj, oldStats.get(statsObj.getColName()));
+ }
+
+ // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
Review Comment:
is this even planned or we can just drop the comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546289389
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryingExecutor.java:
##########
@@ -95,22 +95,26 @@ public T run() throws MetaException {
LOG.error(msg, e1);
throw new MetaException(msg + e1.getMessage());
}
+ LOG.debug("Exception occurred in running: {}", commandName, t);
}
}
}
- private void checkException(Exception e) throws MetaException {
- if (retryPolicy != null && !retryPolicy.test(e)) {
+ private Throwable checkException(Throwable e) throws MetaException {
+ Throwable cause = e;
+ if (e instanceof InvocationTargetException ||
Review Comment:
could we use ExceptionUtils.getRootCause(e) here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546602303
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryingExecutor.java:
##########
@@ -95,22 +95,26 @@ public T run() throws MetaException {
LOG.error(msg, e1);
throw new MetaException(msg + e1.getMessage());
}
+ LOG.debug("Exception occurred in running: {}", commandName, t);
}
}
}
- private void checkException(Exception e) throws MetaException {
- if (retryPolicy != null && !retryPolicy.test(e)) {
+ private Throwable checkException(Throwable e) throws MetaException {
+ Throwable cause = e;
+ if (e instanceof InvocationTargetException ||
Review Comment:
The `cause` will be printed before the next `retry`, I think we'd better to show a clean stack trace of the `cause`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3541272164
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=ACCEPTED)
Measures
 [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546317274
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
Review Comment:
should it be a global config, we can't change it anyways on session level, don't we?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546300464
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -11080,8 +11093,10 @@ public List<WriteEventInfo> getAllWriteEventInfo(long txnId, String dbName, Stri
return writeEventInfoList;
}
- private void executePlainSQL(String sql, Consumer<Exception> exceptionConsumer)
- throws SQLException {
+ private void executePlainSQL(String sql,
+ boolean atLeastOnRecord,
Review Comment:
typo: `atLeastOneRecord` or maybe `atLeastOneRow`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546300464
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -11080,8 +11093,10 @@ public List<WriteEventInfo> getAllWriteEventInfo(long txnId, String dbName, Stri
return writeEventInfoList;
}
- private void executePlainSQL(String sql, Consumer<Exception> exceptionConsumer)
- throws SQLException {
+ private void executePlainSQL(String sql,
+ boolean atLeastOnRecord,
Review Comment:
typo: `atLeastOneRecord`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 merged PR #6188:
URL: https://github.com/apache/hive/pull/6188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546258581
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -3608,7 +3609,7 @@ protected Integer getJdoResult(GetHelper<Integer> ctx)
* you want results for. E.g., if resultsCol is partitionName, the Collection
* has types of String, and if resultsCol is null, the types are MPartition.
*/
- private Collection<String> getPartitionPsQueryResults(String catName, String dbName,
+ private <T> Collection<T> getPartitionPsQueryResults(String catName, String dbName,
Review Comment:
why use generic type? aren't we always return MPartition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546300464
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -11080,8 +11093,10 @@ public List<WriteEventInfo> getAllWriteEventInfo(long txnId, String dbName, Stri
return writeEventInfoList;
}
- private void executePlainSQL(String sql, Consumer<Exception> exceptionConsumer)
- throws SQLException {
+ private void executePlainSQL(String sql,
+ boolean atLeastOnRecord,
Review Comment:
typo: atLeastOneRecord
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3532915909
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=ACCEPTED)
Measures
 [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3538979051
I would suggest to revert the original PR and keep the fix in 1 commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3540730491
> I'm trying to keep the changes to be reviewed simple, and avoid a next cycle of reviews on the old one. This part changes the identity-type of MPartition in the package.jdo, when porting it requires a bit consideration:
sorry, i thought it's a fix for recent commit (updateTableColumnStatistics), but now i see it's an incremental change (updatePartitionColumnStatistics)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546296468
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java:
##########
@@ -1102,6 +1102,7 @@ Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats, Strin
* @throws InvalidObjectException the stats object is invalid
* @throws InvalidInputException unable to record the stats for the table
*/
+ @Deprecated
Review Comment:
could we add clarification what should be used instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547725496
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryingExecutor.java:
##########
@@ -95,22 +95,26 @@ public T run() throws MetaException {
LOG.error(msg, e1);
throw new MetaException(msg + e1.getMessage());
}
+ LOG.debug("Exception occurred in running: {}", commandName, t);
}
}
}
- private void checkException(Exception e) throws MetaException {
- if (retryPolicy != null && !retryPolicy.test(e)) {
+ private Throwable checkException(Throwable e) throws MetaException {
+ Throwable cause = e;
+ if (e instanceof InvocationTargetException ||
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2547731956
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
Review Comment:
i see you've added them to `User configurable Metastore vars`. How can the user modify it from beeline? is there a hql for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2548183557
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
Review Comment:
A set command could help, e.g, set metaconf:metastore.s4u.nowait.retry.sleep.interval=100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3561011844
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=ACCEPTED)
Measures
 [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546281688
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/RetryingExecutor.java:
##########
@@ -95,22 +95,26 @@ public T run() throws MetaException {
LOG.error(msg, e1);
throw new MetaException(msg + e1.getMessage());
}
+ LOG.debug("Exception occurred in running: {}", commandName, t);
}
}
}
- private void checkException(Exception e) throws MetaException {
- if (retryPolicy != null && !retryPolicy.test(e)) {
+ private Throwable checkException(Throwable e) throws MetaException {
+ Throwable cause = e;
+ if (e instanceof InvocationTargetException ||
+ e instanceof UndeclaredThrowableException) {
+ cause = e.getCause();
+ }
+ if (retryPolicy != null && !retryPolicy.test(cause)) {
String message = "See a fatal exception, avoid to retry the command:" + commandName;
- LOG.info(message, e);
- String errorMessage = ExceptionUtils.getMessage(e);
- if (e instanceof InvocationTargetException || e instanceof UndeclaredThrowableException) {
- errorMessage = ExceptionUtils.getMessage(e.getCause());
- }
+ LOG.info(message, cause);
Review Comment:
should we log here under error?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #6188:
URL: https://github.com/apache/hive/pull/6188#discussion_r2546606280
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
##########
@@ -9246,85 +9249,95 @@ public Map<String, String> updateTableColumnStatistics(ColumnStatistics colStats
}
}
- /**
- * Get partition's column stats
- *
- * @return Map of column name and its stats
- */
- private Map<String, MPartitionColumnStatistics> getPartitionColStats(Table table, String partitionName,
- List<String> colNames, String engine) throws NoSuchObjectException, MetaException {
- Map<String, MPartitionColumnStatistics> statsMap = Maps.newHashMap();
- List<MPartitionColumnStatistics> stats =
- getMPartitionColumnStatistics(table, Lists.newArrayList(partitionName), colNames, engine);
- for (MPartitionColumnStatistics cStat : stats) {
- statsMap.put(cStat.getColName(), cStat);
- }
- return statsMap;
- }
-
@Override
public Map<String, String> updatePartitionColumnStatistics(Table table, MTable mTable, ColumnStatistics colStats,
List<String> partVals, String validWriteIds, long writeId)
throws MetaException, NoSuchObjectException, InvalidObjectException, InvalidInputException {
boolean committed = false;
-
+ long start = System.currentTimeMillis();
+ List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
try {
openTransaction();
- List<ColumnStatisticsObj> statsObjs = colStats.getStatsObj();
- ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
- String catName = statsDesc.isSetCatName() ? statsDesc.getCatName() : getDefaultCatalog(conf);
- Partition partition = convertToPart(catName, statsDesc.getDbName(), statsDesc.getTableName(), getMPartition(
- catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable), TxnUtils.isAcidTable(table));
- List<String> colNames = new ArrayList<>();
-
- for(ColumnStatisticsObj statsObj : statsObjs) {
- colNames.add(statsObj.getColName());
- }
-
- Map<String, MPartitionColumnStatistics> oldStats = getPartitionColStats(table, statsDesc
- .getPartName(), colNames, colStats.getEngine());
-
MPartition mPartition = getMPartition(
catName, statsDesc.getDbName(), statsDesc.getTableName(), partVals, mTable);
- if (partition == null) {
+ if (mPartition == null) {
throw new NoSuchObjectException("Partition for which stats is gathered doesn't exist.");
}
- for (ColumnStatisticsObj statsObj : statsObjs) {
- MPartitionColumnStatistics mStatsObj =
- StatObjectConverter.convertToMPartitionColumnStatistics(mPartition, statsDesc, statsObj, colStats.getEngine());
- writeMPartitionColumnStatistics(table, partition, mStatsObj,
- oldStats.get(statsObj.getColName()));
+ List<String> colNames = new ArrayList<>();
+ for(ColumnStatisticsObj statsObj : statsObjs) {
+ colNames.add(statsObj.getColName());
}
- // TODO: (HIVE-20109) the col stats stats should be in colstats, not in the partition!
- Map<String, String> newParams = new HashMap<>(mPartition.getParameters());
- StatsSetupConst.setColumnStatsState(newParams, colNames);
- boolean isTxn = TxnUtils.isTransactionalTable(table);
- if (isTxn) {
- if (!areTxnStatsSupported) {
- StatsSetupConst.setBasicStatsState(newParams, StatsSetupConst.FALSE);
- } else {
- String errorMsg = verifyStatsChangeCtx(TableName.getDbTable(statsDesc.getDbName(),
- statsDesc.getTableName()),
- mPartition.getParameters(), newParams, writeId, validWriteIds, true);
- if (errorMsg != null) {
- throw new MetaException(errorMsg);
- }
- if (!isCurrentStatsValidForTheQuery(mPartition, validWriteIds, true)) {
- // Make sure we set the flag to invalid regardless of the current value.
+ int maxRetries = MetastoreConf.getIntVar(conf, ConfVars.METASTORE_S4U_NOWAIT_MAX_RETRIES);
Review Comment:
made it a configurable session property, the client can change it without the need to restart
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3559001619
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188) **Quality Gate passed**
Issues
 [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6188&issueStatuses=ACCEPTED)
Measures
 [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6188&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6188&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6188)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29316: Concurrency issue in updatePartitionColumnStatistics [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #6188:
URL: https://github.com/apache/hive/pull/6188#issuecomment-3561789101
Much thx for the review, @deniskuzZ @saihemanth-cloudera!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org