You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "kasakrisz (via GitHub)" <gi...@apache.org> on 2025/05/21 13:34:57 UTC
[PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
kasakrisz opened a new pull request, #5822:
URL: https://github.com/apache/hive/pull/5822
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
* Decouple Compaction Stats Updater from `hive.stats.autogather` setting
* Check stats status when generating the analyze command.
### Why are the changes needed?
`hive.stats.autogather` does not guarantee that stats are maintained by DML statements hence it is not identical with the statement
```
analyze table <table name> compute statistics for columns <column list>;
```
There are various reasons when executing a DML statement the stats become stale. Example: delete statements mark the column stats stale because the delta stats can not be calculated from the deleted records.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
mvn test -Dtest=TestCrudCompactorOnTez#testMajorCompactionUpdateMissingColumnStats -pl itests/hive-unit -Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
would it work for the partitioned table?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
see
````
if (!StatsUtils.areBasicStatsUptoDateForQueryAnswering(part.getTable(), part.getParameters())) {
Logger.debug("Stats for part : " + part.getSpec() + " are not up to date.");
return null;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102410586
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
+ .collect(Collectors.toList());
if (!columnList.isEmpty()) {
sb.append(" for columns ").append(String.join(",", columnList));
+ } else if (StatsSetupConst.areBasicStatsUptoDate(tableProperties)) {
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
would it work for the partitioned table?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe us `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
or better create util method `StatsSetupConst.areColumnStatsUptoDate`?
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
or better create util method `StatsSetupConst.areColumnStatsUptoDate`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5822:
URL: https://github.com/apache/hive/pull/5822#issuecomment-2898244407
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5822) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5822&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5822&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5822&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5822&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5822&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5822)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102564987
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
> Maybe use allMatch
With the filter
```
.filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
```
only the columns without stats collected and updated.
> Would it work for the partitioned table ?
Thanks for mentioning this; added partition support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe us `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
Maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
Would it work for the partitioned table ?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
see
````
if (!StatsUtils.areBasicStatsUptoDateForQueryAnswering(part.getTable(), part.getParameters())) {
Logger.debug("Stats for part : " + part.getSpec() + " are not up to date.");
return null;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
Maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
Would it work for the partitioned table ?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
should we get the partition info ``Partition p = resolvePartition(ci);``
see
````
if (!StatsUtils.areBasicStatsUptoDateForQueryAnswering(part.getTable(), part.getParameters())) {
Logger.debug("Stats for part : " + part.getSpec() + " are not up to date.");
return null;
}
````
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
Maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
Would it work for the partitioned table ?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
should we get the partition info:
``Partition p = resolvePartition(ci);``
see
````
if (!StatsUtils.areBasicStatsUptoDateForQueryAnswering(part.getTable(), part.getParameters())) {
Logger.debug("Stats for part : " + part.getSpec() + " are not up to date.");
return null;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2104396974
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java:
##########
@@ -3665,4 +3670,41 @@ public void testFallbackForMergeCompactionWhenDeleteDeltaPresent() throws Except
verify(primary.get(), times(1)).run(any());
verify(secondary.get(), times(1)).run(any());
}
+
+ @Test
+ public void testMajorCompactionUpdateMissingColumnStats() throws Exception {
+ executeStatementOnDriver("create table " + TABLE1 + "(a int, b varchar(128), c float) stored as orc TBLPROPERTIES " +
Review Comment:
call `drop if exists` first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102564987
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
> Maybe use allMatch
With the filter
```
.filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
```
only the columns without stats collected and updated.
> Would it work for the partitioned table ?
Thanks for mentioning this; added partition support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
would it work for the partitioned table?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
see
````
if (!StatsUtils.areBasicStatsUptoDateForQueryAnswering(part.getTable(), part.getParameters())) {
Logger.debug("Stats for part : " + part.getSpec() + " are not up to date.");
return null;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
and extract to the util method `StatsSetupConst.areColumnStatsUptoDate`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5822:
URL: https://github.com/apache/hive/pull/5822#issuecomment-2911413665
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5822) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5822&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5822&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5822&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5822&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5822&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5822)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
Maybe use `allMatch`
````
boolean isStatsUpToDate = [columnList].stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
Would it work for the partitioned table ?
`StatsSetupConst.areColumnStatsUptoDate(tableProperties`
should we ``Partition p = resolvePartition(ci);``
see
````
if (!StatsUtils.areBasicStatsUptoDateForQueryAnswering(part.getTable(), part.getParameters())) {
Logger.debug("Stats for part : " + part.getSpec() + " are not up to date.");
return null;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102566421
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
+ .collect(Collectors.toList());
if (!columnList.isEmpty()) {
sb.append(" for columns ").append(String.join(",", columnList));
+ } else if (StatsSetupConst.areBasicStatsUptoDate(tableProperties)) {
Review Comment:
Added partition support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz merged PR #5822:
URL: https://github.com/apache/hive/pull/5822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102782980
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
> only the columns without stats collected and updated.
oh, I missed that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5822:
URL: https://github.com/apache/hive/pull/5822#discussion_r2102379019
##########
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/StatsUpdater.java:
##########
@@ -81,10 +83,14 @@ public void gatherStats(HiveConf hiveConf, CompactionInfo ci, Map<String, String
sb.append(")");
}
sb.append(" compute statistics");
- if (!conf.getBoolVar(HiveConf.ConfVars.HIVE_STATS_AUTOGATHER) && ci.isMajorCompaction()) {
- List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci));
+ if (ci.isMajorCompaction()) {
+ List<String> columnList = msc.findColumnsWithStats(CompactionInfo.compactionInfoToStruct(ci)).stream()
+ .filter(columnName -> !StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))
Review Comment:
maybe
````
boolean isStatsUpToDate = .. `columnList.stream.allMatch(columnName -> StatsSetupConst.areColumnStatsUptoDate(tableProperties, columnName))`
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28960: Compaction Stats updater does not collect column stats when hive.stats.autogather is true [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5822:
URL: https://github.com/apache/hive/pull/5822#issuecomment-2901484717
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5822) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5822&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5822&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5822&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5822&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5822&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5822)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org