You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "thomasrebele (via GitHub)" <gi...@apache.org> on 2025/11/03 20:01:32 UTC
[PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
thomasrebele opened a new pull request, #6163:
URL: https://github.com/apache/hive/pull/6163
### What changes were proposed in this pull request?
I've updated the metastore dump with the export from a TPC-DS 30TB database with histogram. I've also regenerated the qfiles with org.apache.hadoop.hive.cli.TestTezTPCDS30TBPerfCliDriver.
### Why are the changes needed?
The PR makes it possible to test the planning phase with histograms.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Execute TestTezTPCDS30TBPerfCliDriver locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3904584785
@thomasrebele https://github.com/apache/hive/pull/6177 is closed, so we can't merge this PR, is that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3489880345
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6163) **Quality Gate passed**
Issues
 [22 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6163&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6163&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6163&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6163&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6163&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6163)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele closed pull request #6163: HIVE-26830: Update TPCDS30TB metastore dump with histograms
URL: https://github.com/apache/hive/pull/6163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3492582155
I've investigated the changes to the q.out file for `ql/src/test/queries/clientpositive/perf/query96.q`. The NUM_DISTINCT varies slightly between the old and the new metastore dump (due to the randomness of the HLL algorithm). That influences the selectivity of certain predicates, which lead to different row count estimations. Sometimes they change enough to influence the join order.
The new metastore is an improvement. Besides the histogram statistics, it fixes the statistics for NUM_NULLS, which are very often 0 in the old dump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3868703258
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3485230760
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6163) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6163&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6163&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6163&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6163&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6163&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6163)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3492033838
omg 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3913306332
The PR https://github.com/apache/hive/pull/6177 has been superseded by https://github.com/apache/hive/pull/6194. However, the histogram statistics would need to be regenerated. A new PR can be opened once there's an updated version of the metastore dump.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3482614094
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6163) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6163&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6163&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6163&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6163&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6163&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6163)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-26830: Update TPCDS30TB metastore dump with histograms [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on PR #6163:
URL: https://github.com/apache/hive/pull/6163#issuecomment-3520738496
The histogram stats fix [HIVE-29275](https://issues.apache.org/jira/browse/HIVE-29275) changes query 28, 48, and 49. So we better merge https://github.com/apache/hive/pull/6177 first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org