You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "thomasrebele (via GitHub)" <gi...@apache.org> on 2026/04/10 11:16:55 UTC
[PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
thomasrebele opened a new pull request, #6420:
URL: https://github.com/apache/hive/pull/6420
[HIVE-29554](https://issues.apache.org/jira/browse/HIVE-29554)
### What changes were proposed in this pull request?
Extract a common superclass MColumnStatistics from MPartitionColumnStatistics and MColumnStatistics and combine some methods that use the latter two.
### Why are the changes needed?
There is a lot of code duplication around column statistics, which increases the risk of diverging code and partially applied fixes.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
1. I've run Hive from the master branch (13f3208c01fec2d20108302efc3fd033d1d76a19), created a table with some values, and executed `analyze table tab1 compute statistics for columns;`. I verified the statistics with `describe formatted tab1 a;`.
2. I stopped Hive, and run Hive with the changes of the PR. I checked that the statistics are shown correctly. I inserted more values into the table and executed the ANALYZE TABLE ... command. I verified the statistics.
3. I stopped Hive, and run Hive from the master branch again. I verified the statistics. I inserted more values into the table and executed the ANALYZE TABLE ... command. I verified the statistics.
I started and run the DESCRIBE FORMATTED, ANALYZE TABLE, DESCRIBE FORMATTED sequence for
* a metastore server with the changes from the PR, and
* a hiveserver and beeline from the master branch
and vice versa, i.e.,
* a metastore server from the master branch, and
* a hiveserver and beeline with the changes from the PR
All commands worked as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6420:
URL: https://github.com/apache/hive/pull/6420#discussion_r3073721363
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MColumnStatistics.java:
##########
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.model;
+
+public abstract class MColumnStatistics {
+ private String colName;
+ private String colType;
+ private String engine;
+
+ private Long longLowValue;
+ private Long longHighValue;
+ private Double doubleLowValue;
+ private Double doubleHighValue;
+ private String decimalLowValue;
+ private String decimalHighValue;
+ private Long numNulls;
+ private Long numDVs;
+ private byte[] bitVector = new byte[] { 'H', 'L' };
+ private byte[] histogram = new byte[] {};
+ private Double avgColLen;
+ private Long maxColLen;
+ private Long numTrues;
+ private Long numFalses;
+ private long lastAnalyzed;
+
+ public String getColName() {
+ return colName;
+ }
+
+ public void setColName(String colName) {
+ this.colName = colName;
+ }
+
+ public String getColType() {
+ return colType;
+ }
+
+ public void setColType(String colType) {
+ this.colType = colType;
+ }
+
+ public Long getNumNulls() {
+ return numNulls;
+ }
+
+ public void setNumNulls(long numNulls) {
+ this.numNulls = numNulls;
+ }
+
+ public Long getNumDVs() {
+ return numDVs;
+ }
+
+ public void setNumDVs(long numDVs) {
+ this.numDVs = numDVs;
+ }
+
+ public Double getAvgColLen() {
+ return avgColLen;
+ }
+
+ public void setAvgColLen(double avgColLen) {
+ this.avgColLen = avgColLen;
+ }
+
+ public Long getMaxColLen() {
+ return maxColLen;
+ }
+
+ public void setMaxColLen(long maxColLen) {
+ this.maxColLen = maxColLen;
+ }
+
+ public Long getNumTrues() {
+ return numTrues;
+ }
+
+ public void setNumTrues(long numTrues) {
+ this.numTrues = numTrues;
+ }
+
+ public Long getNumFalses() {
+ return numFalses;
+ }
+
+ public void setNumFalses(long numFalses) {
+ this.numFalses = numFalses;
+ }
+
+ public long getLastAnalyzed() {
+ return lastAnalyzed;
+ }
+
+ public void setLastAnalyzed(long lastAnalyzed) {
+ this.lastAnalyzed = lastAnalyzed;
+ }
+
+ public void setBooleanStats(Long numTrues, Long numFalses, Long numNulls) {
+ this.numTrues = numTrues;
+ this.numFalses = numFalses;
+ this.numNulls = numNulls;
+ }
+
+ public void setLongStats(Long numNulls, Long numNDVs, byte[] bitVector, byte[] histogram,
+ Long lowValue, Long highValue) {
+ this.numNulls = numNulls;
+ this.numDVs = numNDVs;
+ setBitVector(bitVector);
+ setHistogram(histogram);
+ this.longLowValue = lowValue;
+ this.longHighValue = highValue;
+ }
+
+ public void setDoubleStats(Long numNulls, Long numNDVs, byte[] bitVector, byte[] histogram,
+ Double lowValue, Double highValue) {
+ this.numNulls = numNulls;
+ this.numDVs = numNDVs;
+ setBitVector(bitVector);
+ setHistogram(histogram);
+ this.doubleLowValue = lowValue;
+ this.doubleHighValue = highValue;
+ }
+
+ public void setDecimalStats(Long numNulls, Long numNDVs, byte[] bitVector, byte[] histogram,
+ String lowValue, String highValue) {
+ this.numNulls = numNulls;
+ this.numDVs = numNDVs;
+ setBitVector(bitVector);
+ setHistogram(histogram);
+ this.decimalLowValue = lowValue;
+ this.decimalHighValue = highValue;
+ }
+
+ public void setStringStats(Long numNulls, Long numNDVs, byte[] bitVector, Long maxColLen, Double avgColLen) {
+ this.numNulls = numNulls;
+ this.numDVs = numNDVs;
+ setBitVector(bitVector);
+ this.maxColLen = maxColLen;
+ this.avgColLen = avgColLen;
+ }
+
+ public void setBinaryStats(Long numNulls, Long maxColLen, Double avgColLen) {
+ this.numNulls = numNulls;
+ this.maxColLen = maxColLen;
+ this.avgColLen = avgColLen;
+ }
+
+ public void setDateStats(Long numNulls, Long numNDVs, byte[] bitVector, byte[] histogram, Long lowValue, Long highValue) {
Review Comment:
There are Sonar warnings about the length of the line. The original methods in MPartitionColumnStatistics and MTableColumnStatistics had them as well. In case I make an update to the PR, I'll fix it, otherwise I would just leave that line (and `setTimestampStats`) as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 merged PR #6420:
URL: https://github.com/apache/hive/pull/6420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6420:
URL: https://github.com/apache/hive/pull/6420#issuecomment-4225667696
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6420) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6420&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6420&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6420&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6420&metric=new_coverage&view=list)
 [1.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6420&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6420)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6420:
URL: https://github.com/apache/hive/pull/6420#issuecomment-4236354855
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6420) **Quality Gate passed**
Issues
 [14 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6420&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6420&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6420&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6420&metric=new_coverage&view=list)
 [1.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6420&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6420)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6420:
URL: https://github.com/apache/hive/pull/6420#discussion_r3073738419
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java:
##########
Review Comment:
Sonar warns about the "cognitive complexity" limit. The limit has been exceeded in the original implementation as well. I think refactoring the methods is out-of-scope for this PR, so I'll just leave them as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29554: Refactor MTableColumnStatistics and MPartitionColumnStatistics to avoid code duplication [hive]
Posted by "thomasrebele (via GitHub)" <gi...@apache.org>.
thomasrebele commented on code in PR #6420:
URL: https://github.com/apache/hive/pull/6420#discussion_r3073738419
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java:
##########
Review Comment:
Sonar warns about the "cognitive complexity" limit. I think refactoring the methods is out-of-scope for this PR, so I'll just leave them as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org