You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "simhadri-g (via GitHub)" <gi...@apache.org> on 2023/06/18 00:13:14 UTC
[GitHub] [hive] simhadri-g opened a new pull request, #4431: [WIP - do not review] HIVE-27448: Hive Iceberg: Merge column stats
simhadri-g opened a new pull request, #4431:
URL: https://github.com/apache/hive/pull/4431
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### Is the change a dependency upgrade?
<!--
If yes, please attach a file with output from mvn dependency:tree to validate a complete upgrade of dependency.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1600015969
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1245092897
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
+ return false;
+ }
+
+ private void checkAndMergeStats(ColumnStatistics statsObjNew, Table tbl,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable, ColumnStatsDesc columnStatsDesc) {
+ Long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (previousSnapshotId != null && canProvideColStatistics(tbl, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId));
+ if (statsObjOld != null && statsObjOld.getStatsObjSize() != 0 && !statsObjNew.getStatsObj().isEmpty()) {
+ try {
+ MetaStoreServerUtils.mergeColStats(statsObjNew, statsObjOld);
+ StatsSetupConst.setColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
+ } catch (InvalidObjectException e) {
+ StatsSetupConst.removeColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
Review Comment:
fixed, we will persist stats only after the successful merge and return true. If unsuccessful not write the stats and returns false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1649639347
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1270678716
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
sure, thanks @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273319840
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
@simhadri-g, please add a link to a follow-up ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243545825
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -429,45 +434,50 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
ImmutableMap.of()));
writer.finish();
} catch (IOException e) {
- LOG.error(String.valueOf(e));
+ LOG.error("Unable to write stats to puffin file", e.getMessage());
}
return false;
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ if (fs.exists(statsPath)) {
Review Comment:
simply return result of fs.exists(statsPath)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239508824
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
Review Comment:
Done, split into checkAndMergeStats and CheckAndInvalidateStats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267229037
##########
ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java:
##########
@@ -221,7 +222,7 @@ public int persistColumnStats(Hive db, Table tbl) throws HiveException, MetaExce
start = System. currentTimeMillis();
if (tbl != null && tbl.isNonNative() && tbl.getStorageHandler().canSetColStatistics(tbl)) {
- boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats);
+ boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats, colStatDesc);
Review Comment:
sure, removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267227882
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1269307040
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
I did a quick search and `bit_vector` has `HL` in case of non iceberg tables in our q test outs when the column stats present for that column.
@simhadri-g Could you please look into why was it removed in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1648203692
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ merged pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #4431:
URL: https://github.com/apache/hive/pull/4431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267288905
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
I suspect it is removed here because of:
[HIVE-23846](https://issues.apache.org/jira/browse/HIVE-23846)
Avoid unnecessary serialization and deserialization of bitvectors
> In the method getNdvEstimator of <datatype>ColumnStatsDataInspector, it will call isSetBitVectors(), in which it serializes the bitvectors again when we already have deserialized bitvectors ndvEstimator.
ps: HIVE-17286 introduced HL to avoid expensive string serialisation for ndv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263503804
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
Review Comment:
canProvideColStatistics -> canProvideColStats
public methods -> ColStatistics
private -> ColStats
to be consistent in namings
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1265161254
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Review Comment:
It was needed to set col stats as inaccurate if the merge fails.
We need it for the next PR for writing stats for individual column blobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243539001
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
+ return false;
+ }
+
+ private void checkAndMergeStats(ColumnStatistics statsObjNew, Table tbl,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable, ColumnStatsDesc columnStatsDesc) {
+ Long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (previousSnapshotId != null && canProvideColStatistics(tbl, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId));
+ if (statsObjOld != null && statsObjOld.getStatsObjSize() != 0 && !statsObjNew.getStatsObj().isEmpty()) {
+ try {
+ MetaStoreServerUtils.mergeColStats(statsObjNew, statsObjOld);
+ StatsSetupConst.setColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
+ } catch (InvalidObjectException e) {
+ StatsSetupConst.removeColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
Review Comment:
wait, we persist stats only after the successful merge, then why do we need `removeColumnStatsState`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1600850033
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1601870573
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238466987
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
} catch (IOException e) {
LOG.error("Error when trying to read iceberg col stats from puffin files: ", e);
+ return null;
Review Comment:
should we return an empty?
````
ColumnStatistics empty = new ColumnStatistics();
csNew.setStatsObj(Lists.newArrayList());
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1630655187
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [28 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273242610
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,13 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
Review Comment:
why not simply
````
colStats.addAll(table.getStorageHandler().getColStatistics(table));
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273319777
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
A alter table command is run during the doCommit Stage, which seems to unnecessarily overwrite the bit_vector . This should be fixed for iceberg tables. I have created follow up jira to address it.
https://issues.apache.org/jira/browse/HIVE-27528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239735013
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,17 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (colStatistics != null) {
+ colStats.addAll(colStatistics);
+ }
+
+ } else {
+ colStats.addAll(context.getDb()
Review Comment:
done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (canProvideColStatistics(hmsTable, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId).toString());
+ if (statsObjOld != null) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243534351
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,15 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (!colStatistics.isEmpty()) {
Review Comment:
why is this if check needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238466987
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
} catch (IOException e) {
LOG.error("Error when trying to read iceberg col stats from puffin files: ", e);
+ return null;
Review Comment:
should we return an empty stats?
````
ColumnStatistics empty = new ColumnStatistics();
csNew.setStatsObj(Lists.newArrayList());
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1602427212
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [17 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238432254
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
should we have a safe check for position indexes? what if the stats file would be corrupted or empty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238434677
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Review Comment:
try to avoid loading table multiple times, you already have it in https://github.com/apache/hive/pull/4431/files#diff-93864ecf035fe51b92185015da842a56837cea89064813de39c278c6f8fed03cR417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239514810
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (canProvideColStatistics(hmsTable, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId).toString());
+ if (statsObjOld != null) {
+ try {
+ MetaStoreServerUtils.mergeColStats(statsObjNew, statsObjOld);
+ } catch (InvalidObjectException e) {
+ LOG.error("Unable to merge column stats: ", e.getMessage());
Review Comment:
Yes, I updated the PR to mark the stats as inaccurate in the catch block.
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,17 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (colStatistics != null) {
Review Comment:
fixed, it now returns an empty list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239739493
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -414,7 +416,7 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
List<ColumnStatistics> colStats) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
+ checkAndMergeOrInvalidateStats(colStats.get(0), hmsTable);
Review Comment:
For now, it will be size one implying all the column stats are stored as a single blob.
But in the next PR when we implement retrieving stats for particular columns this will be modified to store multiple blobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1242181336
##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergColStats.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.List;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestHiveIcebergColStats extends HiveIcebergStorageHandlerWithEngineBase {
+
+ @Test
+ public void testStatsWithInsert() {
+ TableIdentifier identifier = TableIdentifier.of("default", "customers");
+
+ shell.setHiveSessionValue(HiveConf.ConfVars.HIVESTATSAUTOGATHER.varname, true);
+ testTables.createTable(shell, identifier.name(), HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ PartitionSpec.unpartitioned(), fileFormat, ImmutableList.of());
+
+ if (testTableType != TestTables.TestTableType.HIVE_CATALOG) {
+ // If the location is set and we have to gather stats, then we have to update the table stats now
+ shell.executeStatement("ANALYZE TABLE " + identifier + " COMPUTE STATISTICS FOR COLUMNS");
+ }
+
+ String insert = testTables.getInsertQuery(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS, identifier, false);
+ shell.executeStatement(insert);
+
+ checkColStat(identifier.name(), "customer_id", true);
+ checkColStatMinMaxDistinctValue(identifier.name(), "customer_id", 0, 2, 3, 0);
+
+ insert = testTables.getInsertQuery(HiveIcebergStorageHandlerTestUtils.OTHER_CUSTOMER_RECORDS_1, identifier, false);
+ shell.executeStatement(insert);
+
+ checkColStat(identifier.name(), "customer_id", true);
+ checkColStatMinMaxDistinctValue(identifier.name(), "customer_id", 0, 5, 6, 0);
+
+ insert = testTables.getInsertQuery(HiveIcebergStorageHandlerTestUtils.OTHER_CUSTOMER_RECORDS_2, identifier, false);
+ shell.executeStatement(insert);
+ checkColStat(identifier.name(), "customer_id", true);
+ checkColStatMinMaxDistinctValue(identifier.name(), "customer_id", 0, 5, 6, 0);
+ }
+
+ private void checkColStat(String tableName, String colName, boolean accurate) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1258351499
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -410,64 +413,69 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ if (!checkAndInvalidateStats(tbl)) {
Review Comment:
could we rename to `invalidateStatsIfExists()`
````
boolean rewriteStats = invalidateStatsIfExists(tbl)
if (!rewriteStats) {
checkAndMergeStats(colStats.get(0), tbl);
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1632386247
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263498080
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
Review Comment:
could we extract this into `writeColStatistics(colStats)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263500111
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean removeStatsIfExists(Table tbl) throws IOException {
Review Comment:
removeStatsIfExists -> removeColStatsIfExists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263500479
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean removeStatsIfExists(Table tbl) throws IOException {
+ Path statsPath = getStatsPath(tbl);
+ FileSystem fs = statsPath.getFileSystem(conf);
+ if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
+ }
+ return false;
+ }
+
+ private void checkAndMergeStats(ColumnStatistics statsObjNew, Table tbl) throws InvalidObjectException {
Review Comment:
checkAndMergeStats -> checkAndMergeColStats
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
Review Comment:
getColStatsPath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1266567251
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ return fs.exists(statsPath);
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
+ try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath.toString())).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
Review Comment:
sorry, my bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1268331766
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
Hive provides it as a list of column stats :
https://github.com/apache/hive/blame/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L201
We are serializing this and writing the stats:
https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L462
Map<BlobMetadata, List< ColumnStatistics >>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1603799508
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238420621
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -437,17 +439,20 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(hmsTable, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, long snapshotId) {
+ Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Review Comment:
you are loading the table twice ignoring with skipCache, see https://github.com/apache/hive/pull/4431/files#diff-93864ecf035fe51b92185015da842a56837cea89064813de39c278c6f8fed03cR441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238440952
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
Review Comment:
this method is overloaded. could we split it on invalidate and merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243542602
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
btw, why do we need to delete, can't we simply overwrite the file with new stats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243557507
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -411,10 +414,12 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
@Override
public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
+ if (!checkAndInvalidateStats(tbl)) {
+ checkAndMergeStats(colStats.get(0), tbl, hmsTable, columnStatsDesc);
Review Comment:
shouldn't we always merge the stats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243764316
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
Puffin writer calls create instead of createoverwrite in PositionOutputStream .
https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/puffin/PuffinWriter.java#L66
As a result, it "throws file already exists exception". So we had to delete and write instead of overwrite.
Or fix in iceberg repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238450232
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (canProvideColStatistics(hmsTable, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId).toString());
+ if (statsObjOld != null) {
+ try {
+ MetaStoreServerUtils.mergeColStats(statsObjNew, statsObjOld);
+ } catch (InvalidObjectException e) {
+ LOG.error("Unable to merge column stats: ", e.getMessage());
Review Comment:
would we mark stats as accurate in case of an exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238423799
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
Review Comment:
what is this equals null check? why are you calling `readColStats` twice??
could we return an empty list instead of null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238438091
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -414,7 +416,7 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
List<ColumnStatistics> colStats) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
+ checkAndMergeOrInvalidateStats(colStats.get(0), hmsTable);
Review Comment:
would it be always size 1? what it it's empty?
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -414,7 +416,7 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
List<ColumnStatistics> colStats) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
+ checkAndMergeOrInvalidateStats(colStats.get(0), hmsTable);
Review Comment:
would it be always size 1? what if it's empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1258376149
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
> @SourabhBadhya, why do we need per column Merge result?
@deniskuzZ If there is a query which does Alter table Change column datatype.
The column stats for that given column written by the older snapshots is of a different type. Hence they cant be merged. But other columns can proceed with merging the column stats (at least this seems to be the current behaviour for native tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263498965
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ return fs.exists(statsPath);
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
Review Comment:
readColStats -> readColStatistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263498080
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
Review Comment:
could we extract this into `writeColStats(colStats)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1647823080
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267288905
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
I suspect it is removed here because of:
[HIVE-23846](https://issues.apache.org/jira/browse/HIVE-23846) : Avoid unnecessary serialization and deserialization of bitvectors
> In the method getNdvEstimator of <datatype>ColumnStatsDataInspector, it will call isSetBitVectors(), in which it serializes the bitvectors again when we already have deserialized bitvectors ndvEstimator.
ps: HIVE-17286 introduced HL to avoid expensive string serialisation for ndv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273319400
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
````
The bit vector that contains the ndv values is being overwritten by updatecolstats which is called by alter table command.
That's what I want to fix in the alter Table PR
Since bit vector is being overwritten, Hyper log log is not being used
So HL is missing
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273320317
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
https://issues.apache.org/jira/browse/HIVE-27528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1265160821
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
Review Comment:
done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
Review Comment:
Done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ return fs.exists(statsPath);
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
+ try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath.toString())).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
Review Comment:
It is a map. Currently, we are writing all the column stats to a single blob as a result it is in get(0). As we add individual blobs per column this will change.
##########
ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java:
##########
@@ -221,7 +222,7 @@ public int persistColumnStats(Hive db, Table tbl) throws HiveException, MetaExce
start = System. currentTimeMillis();
if (tbl != null && tbl.isNonNative() && tbl.getStorageHandler().canSetColStatistics(tbl)) {
- boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats);
+ boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats, colStatDesc);
Review Comment:
Yes, but we will need it for the next PR for individual columns .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267228315
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,15 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (!colStatistics.isEmpty()) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267241531
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
We first get the blob metadata:
This is a List that will contain all the blobs. Currently we are writing all stats to only one blob. Therefore we get the blobmetadata at index 0;
Next using this blobmetadata info as key , we get the the array list. Currently its just one element in this list.

Map<BlobMetadata, List<ColumnStatistics>>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238460404
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (canProvideColStatistics(hmsTable, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId).toString());
+ if (statsObjOld != null) {
Review Comment:
should extend the condition to
````
if (statsObjOld != null && statsObjOld.getStatsObjSize() != 0 && !statsObjNew.getStatsObj().isEmpty()) {
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238443409
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
Review Comment:
move it 1 line up and check var in the if block
````
if (previousSnapshotId != null && canProvideColStatistics(hmsTable, previousSnapshotId)) {
...
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243532870
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
we can't just change the public API by decommissioning an older signature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1245093415
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
> should it be a warn or error?
Changed it to warn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1648418331
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273336476
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +454,74 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ return writeColStats(colStats, tbl, snapshotId);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1269298406
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
oh, ok. just wonder why we get a list if we just use 1 element
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1640966375
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243546915
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -429,45 +434,50 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
ImmutableMap.of()));
writer.finish();
} catch (IOException e) {
- LOG.error(String.valueOf(e));
+ LOG.error("Unable to write stats to puffin file", e.getMessage());
}
return false;
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ if (fs.exists(statsPath)) {
+ return true;
}
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
+ try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath.toString())).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
} catch (IOException e) {
LOG.error("Error when trying to read iceberg col stats from puffin files: ", e);
Review Comment:
should it be `error` or `warn`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243531651
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
should it be a `warn` or `error`? even if we failed to execute `delete`/`invalidate` should we try to merge the stats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1637943657
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263505595
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Review Comment:
is it even used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239507756
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -437,17 +439,20 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(hmsTable, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, long snapshotId) {
+ Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239331586
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
+ return false;
+ }
+
+ private void checkAndMergeStats(ColumnStatistics statsObjNew, Table tbl,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable, ColumnStatsDesc columnStatsDesc) {
+ Long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (previousSnapshotId != null && canProvideColStatistics(tbl, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId));
+ if (statsObjOld != null && statsObjOld.getStatsObjSize() != 0 && !statsObjNew.getStatsObj().isEmpty()) {
+ try {
+ MetaStoreServerUtils.mergeColStats(statsObjNew, statsObjOld);
+ StatsSetupConst.setColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
+ } catch (InvalidObjectException e) {
+ StatsSetupConst.removeColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
Review Comment:
Why do we need to set or remove the column stats state here? I think setting it in a single place depending upon whether statistics is correctly written as done in #4440 should suffice right?
##########
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergColStats.java:
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.util.List;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestHiveIcebergColStats extends HiveIcebergStorageHandlerWithEngineBase {
+
+ @Test
+ public void testStatsWithInsert() {
+ TableIdentifier identifier = TableIdentifier.of("default", "customers");
+
+ shell.setHiveSessionValue(HiveConf.ConfVars.HIVESTATSAUTOGATHER.varname, true);
+ testTables.createTable(shell, identifier.name(), HiveIcebergStorageHandlerTestUtils.CUSTOMER_SCHEMA,
+ PartitionSpec.unpartitioned(), fileFormat, ImmutableList.of());
+
+ if (testTableType != TestTables.TestTableType.HIVE_CATALOG) {
+ // If the location is set and we have to gather stats, then we have to update the table stats now
+ shell.executeStatement("ANALYZE TABLE " + identifier + " COMPUTE STATISTICS FOR COLUMNS");
+ }
+
+ String insert = testTables.getInsertQuery(HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS, identifier, false);
+ shell.executeStatement(insert);
+
+ checkColStat(identifier.name(), "customer_id", true);
+ checkColStatMinMaxDistinctValue(identifier.name(), "customer_id", 0, 2, 3, 0);
+
+ insert = testTables.getInsertQuery(HiveIcebergStorageHandlerTestUtils.OTHER_CUSTOMER_RECORDS_1, identifier, false);
+ shell.executeStatement(insert);
+
+ checkColStat(identifier.name(), "customer_id", true);
+ checkColStatMinMaxDistinctValue(identifier.name(), "customer_id", 0, 5, 6, 0);
+
+ insert = testTables.getInsertQuery(HiveIcebergStorageHandlerTestUtils.OTHER_CUSTOMER_RECORDS_2, identifier, false);
+ shell.executeStatement(insert);
+ checkColStat(identifier.name(), "customer_id", true);
+ checkColStatMinMaxDistinctValue(identifier.name(), "customer_id", 0, 5, 6, 0);
+ }
+
+ private void checkColStat(String tableName, String colName, boolean accurate) {
Review Comment:
Probably move this function to the base class so that TestHiveIcebergStatistics can also use the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1247721559
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -410,64 +413,69 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
Review Comment:
Since we are changing the API, can we think of returning a map<Column, Boolean Result> OR a generic object containing mapping of (Column Name -> Merge result) (since we cant predict things that this generic object will contain).
This return type is useful in the case of Alter table Change column's datatype.
@simhadri-g @deniskuzZ WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1266569498
##########
ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java:
##########
@@ -221,7 +222,7 @@ public int persistColumnStats(Hive db, Table tbl) throws HiveException, MetaExce
start = System. currentTimeMillis();
if (tbl != null && tbl.isNonNative() && tbl.getStorageHandler().canSetColStatistics(tbl)) {
- boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats);
+ boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats, colStatDesc);
Review Comment:
why don't we introduce it there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263514080
##########
ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java:
##########
@@ -221,7 +222,7 @@ public int persistColumnStats(Hive db, Table tbl) throws HiveException, MetaExce
start = System. currentTimeMillis();
if (tbl != null && tbl.isNonNative() && tbl.getStorageHandler().canSetColStatistics(tbl)) {
- boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats);
+ boolean success = tbl.getStorageHandler().setColStatistics(tbl, colStats, colStatDesc);
Review Comment:
colStatDesc redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263500627
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
Review Comment:
getStatsPath -> getColStatsPath
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ return fs.exists(statsPath);
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
Review Comment:
readColStats -> readColStatistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1267233529
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
.get(<index>) will through IndexOutofBounds exception. This exception is caught and If the file is corrupted or empty we will return empty ColumnStats Object,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1268331766
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
Hive provides it as a list of column stats, we are serializing this: https://github.com/apache/hive/blame/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1641973054
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: [WIP - do not review] HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1595915918
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239509016
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Review Comment:
fixed
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238434677
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Review Comment:
try to avoid loading table multiple times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238452687
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,17 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (colStatistics != null) {
+ colStats.addAll(colStatistics);
+ }
+
+ } else {
+ colStats.addAll(context.getDb()
Review Comment:
fix the formatting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243531651
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
should it be a `warn` or `error`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243764316
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
Puffin writes calls create instead of createoverwrite in PositionOutputStream .
https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/puffin/PuffinWriter.java#L66
As a result, it "throws file already exists exception". So we had to delete and write instead of overwrite.
Or fix in iceberg repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243764316
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
Puffin writer calls create instead of createoverwrite in PositionOutputStream .
https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/puffin/PuffinWriter.java#L66
As a result, it "throws file already exists exception". So we had to delete and write instead of overwrite.
Or we can fix it in iceberg repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1607642287
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1259289961
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
The mergeColStats from the MetaStoreServerUtils is a void method. https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java#L795
Inorder to return (Column Name -> Merge result ) we will have to modify the MetaStoreServerUtils.mergeColStats to return the result for each column or move the set column stats accurate to MetaStoreServerUtils.mergeColStats method.
Since this would require a lot of changes, can we address that in a different jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1259289961
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
The mergeColStats from the MetaStoreServerUtils is a void method. https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java#L795
Inorder to return (Column Name -> Merge result ) we will have to modify the MetaStoreServerUtils.mergeColStats to return the result for each column or move the set column stats accurate to MetaStoreServerUtils.mergeColStats method.
Since this would require a lot of changes , can we please address that in a different jira?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1268143246
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
I still don't get, how adding minor delta stats merge caused that behavior change? @kasakrisz any hints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1268331766
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
Hive provides it as a list of column stats :
https://github.com/apache/hive/blame/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L201
We are serializing this and writing the stats:
https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238432254
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
should we have a safe check for position indexes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238447730
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
Review Comment:
change statsPath type to Path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238468538
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,17 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (colStatistics != null) {
Review Comment:
we shouldn't return null
````
colStats.addAll(
table.getStorageHandler().getColStatistics(table))'
````
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,17 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
+ if (colStatistics != null) {
Review Comment:
we shouldn't return null
````
colStats.addAll(
table.getStorageHandler().getColStatistics(table));
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239508071
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
Review Comment:
fixed.
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
} catch (IOException e) {
LOG.error("Error when trying to read iceberg col stats from puffin files: ", e);
+ return null;
Review Comment:
Agreed, updated the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1258351499
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -410,64 +413,69 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ if (!checkAndInvalidateStats(tbl)) {
Review Comment:
could we rename to `invalidateStatsIfExists()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1258376149
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
> @SourabhBadhya, why do we need per column Merge result?
@deniskuzZ If there is a query which does Alter table Change column datatype.
The column stats for that given column written by the older snapshots is of a different type. Hence they cant be merged. But other columns can proceed with merging the column stats (at least this seems to be the current behaviour for native tables).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243531651
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
should it be a `warn` or `error`? even if we failed to execute `delete` we are trying to merge the stats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243531651
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
Review Comment:
should it be a `warn` or `error`? even if we failed to execute `delete` should we try to merge the stats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243571172
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -411,10 +414,12 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
@Override
public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
+ if (!checkAndInvalidateStats(tbl)) {
+ checkAndMergeStats(colStats.get(0), tbl, hmsTable, columnStatsDesc);
Review Comment:
if it is an `analyze <table> compute statistics for column`, we should not merge the stats but overwrite them with the complete stats. So for analyze table command we should invalidate the stats, here it was done by deleting the older stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1609907682
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4431)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4431&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4431&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] SourabhBadhya commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "SourabhBadhya (via GitHub)" <gi...@apache.org>.
SourabhBadhya commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1247725865
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
Since we are changing the API, can we think of returning a map<Column, Boolean Result> OR a generic object containing mapping of (Column Name -> Merge result) (since we cant predict things that this generic object will contain).
This return type is useful in the case of Alter table Change column's datatype.
@simhadri-g @deniskuzZ WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239735693
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238443409
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,20 +501,47 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
+ }
+
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
}
- private void invalidateStats(Path statsPath) {
+ private void checkAndMergeOrInvalidateStats(ColumnStatistics statsObjNew,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
fs.delete(statsPath, true);
+ } else {
+ // Insert queries
+ mergeStats(statsObjNew, tbl, hmsTable);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
}
+ private void mergeStats(ColumnStatistics statsObjNew, Table tbl, org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+ if (tbl.currentSnapshot().parentId() != null) {
+ long previousSnapshotId = tbl.currentSnapshot().parentId();
Review Comment:
move it 1 line up and check var in the if block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238420621
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -437,17 +439,20 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(hmsTable, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, long snapshotId) {
+ Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
Review Comment:
you are loading the table twice ignoring with skipCache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238423799
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
Review Comment:
what is this equals null check? why are you calling `readColStats` twice??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1238469991
##########
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##########
@@ -96,7 +96,7 @@ avg_col_len 3.4444444444444446
max_col_len 5
num_trues
num_falses
-bit_vector HL
+bit_vector
Review Comment:
why is this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1239746172
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -493,18 +503,42 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean checkAndInvalidateStats(Table tbl) {
+ Path statsPath = getStatsPath(tbl);
try {
FileSystem fs = statsPath.getFileSystem(conf);
if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
}
} catch (IOException e) {
LOG.error("Failed to invalidate stale column stats: {}", e.getMessage());
}
+ return false;
+ }
+
+ private void checkAndMergeStats(ColumnStatistics statsObjNew, Table tbl,
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable, ColumnStatsDesc columnStatsDesc) {
+ Long previousSnapshotId = tbl.currentSnapshot().parentId();
+ if (previousSnapshotId != null && canProvideColStatistics(tbl, previousSnapshotId)) {
+ ColumnStatistics statsObjOld = readColStats(tbl, getStatsPath(tbl, previousSnapshotId));
+ if (statsObjOld != null && statsObjOld.getStatsObjSize() != 0 && !statsObjNew.getStatsObj().isEmpty()) {
+ try {
+ MetaStoreServerUtils.mergeColStats(statsObjNew, statsObjOld);
+ StatsSetupConst.setColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
+ } catch (InvalidObjectException e) {
+ StatsSetupConst.removeColumnStatsState(hmsTable.getParameters(), columnStatsDesc.getColName());
Review Comment:
If there is an exception, merging stats won't complete as a result the stats would be inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243610580
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
It was an oversight, we will need to modify it. We will need the columnStatisticsDesc for the next improvements as well (writing individual blobs per column). Since it is not out in any of the hive releases yet would it be ok to modify the signature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1243610580
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
It was an oversight, we will need to modify it. We will need the columnStatisticsDesc for the next improvements as well. Since it is not out in any of the hive releases yet would it be ok to modify the signature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1265160586
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
Review Comment:
done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean removeStatsIfExists(Table tbl) throws IOException {
Review Comment:
Done
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -536,17 +544,30 @@ private String getStatsSource() {
}
private Path getStatsPath(Table table) {
- return new Path(table.location() + STATS + table.name() + table.currentSnapshot().snapshotId());
+ return getStatsPath(table, table.currentSnapshot().snapshotId());
}
- private void invalidateStats(Path statsPath) {
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- fs.delete(statsPath, true);
+ private Path getStatsPath(Table table, long snapshotId) {
+ return new Path(table.location() + STATS + table.name() + snapshotId);
+ }
+
+ private boolean removeStatsIfExists(Table tbl) throws IOException {
+ Path statsPath = getStatsPath(tbl);
+ FileSystem fs = statsPath.getFileSystem(conf);
+ if (fs.exists(statsPath)) {
+ // Analyze table and stats updater thread
+ return fs.delete(statsPath, true);
+ }
+ return false;
+ }
+
+ private void checkAndMergeStats(ColumnStatistics statsObjNew, Table tbl) throws InvalidObjectException {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263513738
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
sure, added the same comment above, it seems you don't use columnStatisticsDesc obj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1263511062
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +455,70 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ boolean rewriteStats = removeStatsIfExists(tbl);
+ if (!rewriteStats) {
+ checkAndMergeStats(colStats.get(0), tbl);
+ }
+ byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
+ try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
+ .createdBy(Constants.HIVE_ENGINE).build()) {
+ writer.add(new Blob(tbl.name() + "-" + snapshotId, ImmutableList.of(1), tbl.currentSnapshot().snapshotId(),
+ tbl.currentSnapshot().sequenceNumber(), ByteBuffer.wrap(serializeColStats), PuffinCompressionCodec.NONE,
+ ImmutableMap.of()));
+ writer.finish();
+ return true;
+ } catch (IOException e) {
+ LOG.warn("Unable to write stats to puffin file", e.getMessage());
+ return false;
+ }
+ } catch (InvalidObjectException | IOException e) {
+ LOG.warn("Unable to invalidate or merge stats: ", e.getMessage());
return false;
}
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
- }
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ return fs.exists(statsPath);
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
+ try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath.toString())).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
Review Comment:
why do we readAll(blobMetadata), but later on just use `blobMetadata.get(0)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1268138701
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
@simhadri-g, why it's not Map<BlobMetadata, ColumnStatistics>? should we serialize List<ColumnStatisticsObj> instead of ColumnStatistics object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1268331766
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -457,17 +462,20 @@ public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.meta
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String statsPath = getStatsPath(table).toString();
LOG.info("Using stats from puffin file at: {}", statsPath);
+ return readColStats(table, statsPath).equals(null) ? null : readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, String statsPath) {
try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
Review Comment:
Hive provides it as a list of column stats, we are storing this: https://github.com/apache/hive/blame/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1245088566
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -429,45 +434,50 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
ImmutableMap.of()));
writer.finish();
} catch (IOException e) {
- LOG.error(String.valueOf(e));
+ LOG.error("Unable to write stats to puffin file", e.getMessage());
}
return false;
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ if (fs.exists(statsPath)) {
+ return true;
}
+ } catch (IOException e) {
+ LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
+ "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
}
return false;
}
@Override
public List<ColumnStatisticsObj> getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- String statsPath = getStatsPath(table).toString();
+ Path statsPath = getStatsPath(table);
LOG.info("Using stats from puffin file at: {}", statsPath);
- try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath)).build()) {
+ return readColStats(table, statsPath).getStatsObj();
+ }
+
+ private ColumnStatistics readColStats(Table table, Path statsPath) {
+ try (PuffinReader reader = Puffin.read(table.io().newInputFile(statsPath.toString())).build()) {
List<BlobMetadata> blobMetadata = reader.fileMetadata().blobs();
- Map<BlobMetadata, List<ColumnStatistics>> collect =
- Streams.stream(reader.readAll(blobMetadata)).collect(Collectors.toMap(Pair::first,
- blobMetadataByteBufferPair -> SerializationUtils.deserialize(
- ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
- return collect.get(blobMetadata.get(0)).get(0).getStatsObj();
+ Map<BlobMetadata, List<ColumnStatistics>> collect = Streams.stream(reader.readAll(blobMetadata)).collect(
+ Collectors.toMap(Pair::first, blobMetadataByteBufferPair -> SerializationUtils.deserialize(
+ ByteBuffers.toByteArray(blobMetadataByteBufferPair.second()))));
+ return collect.get(blobMetadata.get(0)).get(0);
} catch (IOException e) {
LOG.error("Error when trying to read iceberg col stats from puffin files: ", e);
Review Comment:
Changed it to warn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1245089193
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -429,45 +434,50 @@ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTabl
ImmutableMap.of()));
writer.finish();
} catch (IOException e) {
- LOG.error(String.valueOf(e));
+ LOG.error("Unable to write stats to puffin file", e.getMessage());
}
return false;
}
@Override
public boolean canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
Table table = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
- if (canSetColStatistics(hmsTable)) {
- Path statsPath = getStatsPath(table);
- try {
- FileSystem fs = statsPath.getFileSystem(conf);
- if (fs.exists(statsPath)) {
- return true;
- }
- } catch (IOException e) {
- LOG.warn("Exception when trying to find Iceberg column stats for table:{} , snapshot:{} , " +
- "statsPath: {} , stack trace: {}", table.name(), table.currentSnapshot(), statsPath, e);
+ return canSetColStatistics(hmsTable) && canProvideColStatistics(table, table.currentSnapshot().snapshotId());
+ }
+
+ private boolean canProvideColStatistics(Table table, long snapshotId) {
+ Path statsPath = getStatsPath(table, snapshotId);
+ try {
+ FileSystem fs = statsPath.getFileSystem(conf);
+ if (fs.exists(statsPath)) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1258333588
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -266,7 +267,7 @@ default boolean canProvideBasicStatistics() {
* @return boolean
*/
default boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table table,
- List<ColumnStatistics> colStats) {
+ List<ColumnStatistics> colStats, ColumnStatsDesc columnStatisticsDesc) {
Review Comment:
@SourabhBadhya, why do we need per column Merge result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1259333812
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -410,64 +413,69 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats,
+ ColumnStatsDesc columnStatsDesc) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ try {
+ if (!checkAndInvalidateStats(tbl)) {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4431:
URL: https://github.com/apache/hive/pull/4431#issuecomment-1630418928
Please retry analysis of this Pull-Request directly on [SonarCloud](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4431).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273236346
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -452,65 +454,74 @@ public boolean canSetColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsT
}
@Override
- public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
- List<ColumnStatistics> colStats) {
+ public boolean setColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable, List<ColumnStatistics> colStats) {
Table tbl = IcebergTableUtil.getTable(conf, hmsTable.getTTable());
String snapshotId = String.format("%s-STATS-%d", tbl.name(), tbl.currentSnapshot().snapshotId());
- invalidateStats(getStatsPath(tbl));
- byte[] serializeColStats = SerializationUtils.serialize((Serializable) colStats);
- try (PuffinWriter writer = Puffin.write(tbl.io().newOutputFile(getStatsPath(tbl).toString()))
- .createdBy(Constants.HIVE_ENGINE).build()) {
- writer.add(
- new Blob(
- tbl.name() + "-" + snapshotId,
- ImmutableList.of(1),
- tbl.currentSnapshot().snapshotId(),
- tbl.currentSnapshot().sequenceNumber(),
- ByteBuffer.wrap(serializeColStats),
- PuffinCompressionCodec.NONE,
- ImmutableMap.of()));
- writer.finish();
- return true;
- } catch (IOException e) {
- LOG.error(String.valueOf(e));
+ return writeColStats(colStats, tbl, snapshotId);
Review Comment:
````
return writeColStats(colStats.get(0), tbl, snapshotId);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] simhadri-g commented on a diff in pull request #4431: HIVE-27448: Hive Iceberg: Merge column stats
Posted by "simhadri-g (via GitHub)" <gi...@apache.org>.
simhadri-g commented on code in PR #4431:
URL: https://github.com/apache/hive/pull/4431#discussion_r1273336166
##########
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/desc/DescTableOperation.java:
##########
@@ -208,9 +208,13 @@ private void getColumnDataColPathSpecified(Table table, Partition part, List<Fie
table.setParameters(tableProps);
} else {
cols.addAll(Hive.getFieldsFromDeserializer(desc.getColumnPath(), deserializer, context.getConf()));
- colStats.addAll(
- context.getDb().getTableColumnStatistics(tableName.getDb().toLowerCase(),
- tableName.getTable().toLowerCase(), colNames, false));
+ if (table.isNonNative() && table.getStorageHandler().canProvideColStatistics(table)) {
+ List<ColumnStatisticsObj> colStatistics = table.getStorageHandler().getColStatistics(table);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org