You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "konstantinb (via GitHub)" <gi...@apache.org> on 2026/03/13 06:22:04 UTC

[PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

konstantinb opened a new pull request, #6359:
URL: https://github.com/apache/hive/pull/6359

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
     2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
     3. Ensure you have added or run the appropriate tests for your PR: 
     4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX:  Your PR title ...'.
     5. Be sure to keep the PR description updated to reflect all changes.
     6. Please write your PR title to summarize what this PR proposes.
     7. If possible, provide a concise example to reproduce the issue for a faster review.
   
   -->
   
   ### What changes were proposed in this pull request?
   HIVE-29368: Could this be the truly accurate pessimistic stats combining?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030882052


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/StatEstimator.java:
##########
@@ -39,5 +40,19 @@ public interface StatEstimator {
    * @param argStats the statistics for every argument of the UDF
    * @return {@link ColStatistics} estimate for the actual UDF.
    */
-  public Optional<ColStatistics> estimate(List<ColStatistics> argStats);
+  default Optional<ColStatistics> estimate(List<ColStatistics> argStats) {
+    throw new UnsupportedOperationException("This estimator requires parentStats");
+  }

Review Comment:
   This change eliminates the need to implement estimate(args) for impacted estimators while allowing other estimators that do not need numRows to keep their original estimate(args) implementation, without having to implement estimate(args, numRows). I..e. it felt "cleaner" and allowed to reduce the # of java files than required edits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4114547760

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [20 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [2.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4121260592

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4103872208

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3033018561


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   CASE WHEN statements can appear everywhere in the query plan and the changes in the combiner make this stats better. I understand that there is some correlation with grouping columns but not all queries have a GROUP BY clause thus I feel we could separate the two problems. 
   
   I am not opposed in changing extractNDVGroupingColumns but was thinking that its probably better to do it in a separate PR. If we can't, then option 2 seems more appealing although I couldn't find out why this +1 logic was introduced in the first place.
   
   Overall, I trust your judgement since you spend much more time on the topic so I am happy to let you decide how you want to move forward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4153794528

   > I was unable to find a way to distinguish between "const NULL" ColStatistics objects and objects that truly have an "unknown NDV" without changing the ColStatistics class or refactoring the StatEstimator interface, and making...
   
   @konstantinb Since the previous version of the combiner didn't bother much about unknown NDV values, I am not too opinionated about handling the case where countDistinct is zero. Personally, I would be fine even with something simplistic like the following:
   
   ```java
       if (stat.getCountDistint() >= 0 && result.getCountDistint() >= 0) {
         result.setCountDistint(StatsUtils.safeAdd(result.getCountDistint(), stat.getCountDistint()));
       }
   ```
   From my perspective missing NDVs is an edge case and not something that should appear too often during optimization.
   
   Having said that, I am not against a more elaborate solution like modifying `ColStatistics` or the `StatsEstimator` interface as per previous commits so I will defer the final decision to @konstantinb .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4158960930

   > From my perspective missing NDVs is an edge case and not something that should appear too often during optimization.
   
   @zabetak, over the last few months of struggling with NDVs, I have identified various ways to trigger bad estimates with the current code. The following [estimation change in an .out file](https://github.com/apache/hive/pull/6359#discussion_r3012726545) immediately highlights a 35x underestimation of row count due to treating an NDV of 0 as a "true 0 value".
   
   Too small NDV values can lead to severe underestimation of GROUP BY cardinality; too large NDVs can severely underestimate the cardinality of an IN() filter.
   
   When dealing with very large tables (millions to billions of records), it can become prohibitively expensive to accurately track the number of unique values per column. This really makes missing (unknown) NDV a regular occurrence.
   
   For every attempt to estimate NDV using numRows, I quickly discovered a counterexample of a skewed dataset that led to severe misestimation, causing either a performance problem or an outright query failure. I would be happy to provide those examples if you like.
   
   In some of those cases, a better NDV estimation is possible. For example, with a CASE..WHEN statement with multiple constants. In many other cases, it is genuinely safer to use the most pessimistic algorithm, which handles "unknown NDV" columns surprisingly well.
   
   The very last commit has passing tests. I would appreciate it if you could take another look when you get a chance.
   
   The combiner logic is very close to
   ```
       if (stat.getCountDistint() >= 0 && result.getCountDistint() >= 0) {
         result.setCountDistint(StatsUtils.safeAdd(result.getCountDistint(), stat.getCountDistint()));
       }
   
   ```
   except specific handling NDV==0 for non-constant null columns
   
   cc @okumin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030869220


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   @zabetak thank you for your feedback; I fully understand your concerns and the goal of reducing code changes in one PR.
   
   The main problem with keeping this code 100% intact is that **any** ColStatistics object coming into extractNDVGroupingColumns with an NDV of 0 and numNulls > 0 would be estimated with **1** GROUP BY row count; and this "1" estimate has proven to be the root cause of many severe under-estimations. 
   
   It makes this method logic directly correlated with the proposed changes to PessimisticStatCombiner
   
   Technically, the following two alternatives are feasible alternatives to the current PR code:
   1.
   `        if ((ndv > 0 && cs.getNumNulls() > 0) {
             ndv = StatsUtils.safeAdd(ndv, 1);
           }
   `
   ^^ This one adjusts NDV estimations except for columns with (potentially) unknown NDVs. Applying this change did not cause changes to any already modified .out files **of this branch**. I believe this is the safest possible change for this logic
   
   2. Completely removing the +1 adjustment for NULLs:
   `        ndvValues.add(cs.getCountDistint());
   `
   ^^ This code edit modifies only three files on the current branch
   
   If you firmly oppose any changes to **extractNDVGroupingColumns**, then we will likely need to get back to the drawing board on how to approach this whole problem in a different manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r2988801680


##########
ql/src/test/results/clientpositive/llap/list_bucket_dml_6.q.out:
##########
@@ -1,13 +1,17 @@
-PREHOOK: query: create table list_bucketing_dynamic_part_n3 (key String, value String) 
-    partitioned by (ds String, hr String) 
+hive.merge.mapfiles=true
+hive.merge.mapredfiles=false
+hive.merge.tezfiles=false
+hive.merge.smallfiles.avgsize=16000000

Review Comment:
   this was just a mistake of not regenerating .out file after removing the debug info from the .q file; sorry



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4101575358

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4147041592

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-5111379184

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
   Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4181096433

   @zabetak I've opened a PR https://github.com/apache/hive/pull/6406 as per your suggestion for [HIVE-29534](https://issues.apache.org/jira/browse/HIVE-29534)
   Unfortunately, it triggers a massive set of changes to output files (178 in total)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r2988884907


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/PessimisticStatCombiner.java:
##########
@@ -41,9 +42,14 @@ public void add(ColStatistics stat) {
     if (stat.getAvgColLen() > result.getAvgColLen()) {
       result.setAvgColLen(stat.getAvgColLen());
     }
-    if (stat.getCountDistint() > result.getCountDistint()) {
-      result.setCountDistint(stat.getCountDistint());
+    // NDV=0 is "unknown" only if the stat is NOT a constant.
+    // Constants with NDV=0 (e.g., NULL) are "known zero", not unknown.
+    if ((result.getCountDistint() == 0 && !result.isConst()) || (stat.getCountDistint() == 0 && !stat.isConst())) {
+      result.setCountDistint(0);

Review Comment:
   @zabetak, this is the most complicated problem to solve, in my opinion. The following code: https://github.com/apache/hive/blob/931d4bb62b26de699240c816df439e00644e3dcb/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1634 is very explicit in assigning an NDV of 0 to NULL constants and 1 to non-NULL constants. At the same time, an NDV of "0" for a source column is typically used to indicate that the NDV for the column. is "unknown", which could matter a lot for large tables. Therefore, simply "summing" an NDV of 0 introduces even bigger mis-estimations
   
   To compensate for the "0 NDV" null constant, The following code: https://github.com/apache/hive/blob/931d4bb62b26de699240c816df439e00644e3dcb/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L2113 has a "+1" adjustment when figuring out NDVs of a GROUP BY. 
   
   I am thinking of trying to modify buildColStatForConstant() to treat NULL as a regular constant and seeing if we run into any significant side effects. Thank you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r2986830609


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/ColStatistics.java:
##########
@@ -31,6 +31,7 @@ public class ColStatistics {
   private boolean isPrimaryKey;
   private boolean isEstimated;
   private boolean isFilteredColumn;
+  private boolean isConst;

Review Comment:
   Can we avoid this new indicator? If we add a new field it means that we need to keep them up to date in various places where `ColStatistics` are used so I would prefer to avoid this if possible.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -1578,7 +1578,7 @@ public static ColStatistics getColStatisticsFromExpression(HiveConf conf, Statis
             csList.add(cs);
           }
           if (csList.size() == engfd.getChildren().size()) {
-            Optional<ColStatistics> res = se.estimate(csList);
+            Optional<ColStatistics> res = se.estimate(csList, numRows);

Review Comment:
   A `GenericUDF` can never have an NDV > numRows so instead of putting the responsibility to the implementors of `StatEstimator`  we could enforce the bound here. This would avoid the changes in the existing APIs and make the code more robust since even if the estimator returns an invalid NDV (e.g., greater than the numRows) it wouldn't mess up everything else.  



##########
ql/src/test/results/clientpositive/llap/list_bucket_dml_6.q.out:
##########
@@ -1,13 +1,17 @@
-PREHOOK: query: create table list_bucketing_dynamic_part_n3 (key String, value String) 
-    partitioned by (ds String, hr String) 
+hive.merge.mapfiles=true
+hive.merge.mapredfiles=false
+hive.merge.tezfiles=false
+hive.merge.smallfiles.avgsize=16000000

Review Comment:
   These props shouldn't be here and that's why the CI fails as well. Most likely a bad copy-paster in the wrong place.



##########
ql/src/test/queries/clientpositive/list_bucket_dml_6.q:
##########
@@ -92,6 +94,3 @@ select * from list_bucketing_dynamic_part_n3 where key = '484' and value = 'val_
 select * from list_bucketing_dynamic_part_n3 where key = '484' and value = 'val_484';
 select * from srcpart where ds = '2008-04-08' and key = '484' and value = 'val_484';
 
--- clean up
-drop table list_bucketing_dynamic_part_n3;
-

Review Comment:
   Unrelated changes and whitespace fixes, although minor, increase conflicts and backfor efforts so it's better to keep them apart and avoid them if possible.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/StatEstimator.java:
##########
@@ -24,20 +24,53 @@
 import org.apache.hadoop.hive.ql.plan.ColStatistics;
 
 /**
- * Enables statistics related computation on UDFs
+ * Enables statistics related computation on UDFs.
+ *
+ * <p>This interface provides two default implementations:
+ * <ul>
+ *   <li>{@link #estimate(List)} - clones the first argument's statistics (suitable for most UDFs)</li>
+ *   <li>{@link #estimate(List, long)} - calls estimate(List) and caps NDV at numRows</li>
+ * </ul>
+ *
+ * <p>UDFs that simply pass through statistics (like LOWER, UPPER) can use the defaults.
+ * UDFs that combine statistics (like IF, WHEN, COALESCE) should override {@link #estimate(List)}.
  */
 public interface StatEstimator {
 
   /**
    * Computes the output statistics of the actual UDF.
    *
-   * The estimator should return with a preferably overestimated {@link ColStatistics} object if possible.
-   * The actual estimation logic may decide to not give an estimation; it should return with {@link Optional#empty()}.
+   * <p>The default implementation clones the first argument's statistics, which is suitable
+   * for most UDFs that don't significantly alter the statistical properties of their input.
+   *
+   * <p>Override this method for UDFs that combine multiple inputs (like IF, WHEN, COALESCE)
+   * or significantly transform the data.
+   *
+   * @param argStats the statistics for every argument of the UDF
+   * @return {@link ColStatistics} estimate for the actual UDF, or empty if estimation is not possible
+   */
+  default Optional<ColStatistics> estimate(List<ColStatistics> argStats) {
+    if (argStats.isEmpty()) {
+      return Optional.empty();
+    }
+    return Optional.of(argStats.get(0).clone());
+  }

Review Comment:
   There is no reason to add a default implementation for this method. Leaving things as they are does not break anything.



##########
ql/src/test/queries/clientpositive/ndv_case_const.q:
##########
@@ -0,0 +1,27 @@
+CREATE TABLE t (cond INT, c2 STRING, c100 STRING);

Review Comment:
   As far as I understand this set of tests are mainly targeting the `Statistics: Num rows` entry of the `Group By Operator`. It may be worth adding a comment in here to clarify the intention of these tests so that if/when .q.out changes in the future the reviewer can quickly understand what these tests are supposed to check.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/PessimisticStatCombiner.java:
##########
@@ -41,9 +42,14 @@ public void add(ColStatistics stat) {
     if (stat.getAvgColLen() > result.getAvgColLen()) {
       result.setAvgColLen(stat.getAvgColLen());
     }
-    if (stat.getCountDistint() > result.getCountDistint()) {
-      result.setCountDistint(stat.getCountDistint());
+    // NDV=0 is "unknown" only if the stat is NOT a constant.
+    // Constants with NDV=0 (e.g., NULL) are "known zero", not unknown.
+    if ((result.getCountDistint() == 0 && !result.isConst()) || (stat.getCountDistint() == 0 && !stat.isConst())) {
+      result.setCountDistint(0);

Review Comment:
   Is this special handling here only relevant for NULL constants? For the sake of simplicity could we just skip/ignore stats with unknown NDV? In other words, if NDV is zero just let it be and just proceed with the summation (which effectively is a noop).
   
   As a side note, I am not sure why the count distinct of a NULL constant should be zero and not one. It's just another constant after all.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/StatEstimator.java:
##########
@@ -24,20 +24,53 @@
 import org.apache.hadoop.hive.ql.plan.ColStatistics;
 
 /**
- * Enables statistics related computation on UDFs
+ * Enables statistics related computation on UDFs.
+ *
+ * <p>This interface provides two default implementations:
+ * <ul>
+ *   <li>{@link #estimate(List)} - clones the first argument's statistics (suitable for most UDFs)</li>
+ *   <li>{@link #estimate(List, long)} - calls estimate(List) and caps NDV at numRows</li>
+ * </ul>
+ *
+ * <p>UDFs that simply pass through statistics (like LOWER, UPPER) can use the defaults.
+ * UDFs that combine statistics (like IF, WHEN, COALESCE) should override {@link #estimate(List)}.
  */
 public interface StatEstimator {
 
   /**
    * Computes the output statistics of the actual UDF.
    *
-   * The estimator should return with a preferably overestimated {@link ColStatistics} object if possible.
-   * The actual estimation logic may decide to not give an estimation; it should return with {@link Optional#empty()}.
+   * <p>The default implementation clones the first argument's statistics, which is suitable
+   * for most UDFs that don't significantly alter the statistical properties of their input.
+   *
+   * <p>Override this method for UDFs that combine multiple inputs (like IF, WHEN, COALESCE)
+   * or significantly transform the data.
+   *
+   * @param argStats the statistics for every argument of the UDF
+   * @return {@link ColStatistics} estimate for the actual UDF, or empty if estimation is not possible
+   */
+  default Optional<ColStatistics> estimate(List<ColStatistics> argStats) {
+    if (argStats.isEmpty()) {
+      return Optional.empty();
+    }
+    return Optional.of(argStats.get(0).clone());
+  }
+
+  /**
+   * Computes the output statistics of the actual UDF, ensuring NDV does not exceed numRows.

Review Comment:
   If we want to enforce that NDV does not exceed the numRows then its better to do it outside the estimator than put the burden on each implementation that we may not even have control over.



##########
ql/src/test/queries/clientpositive/list_bucket_dml_6.q:
##########
@@ -1,10 +1,12 @@
 --! qt:dataset:srcpart
+-- this ensures consistent split file counts between localhost & CI runs
+set tez.grouping.split-count=1;

Review Comment:
   This seems to address a flakiness issue that is independent of the PR. If that's the case then better to address it in a separate Jira ticket with a dedicated PR. Mixing flakiness fixes with core changes creates various complications especially during backports to other branches.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/StatEstimator.java:
##########
@@ -24,20 +24,53 @@
 import org.apache.hadoop.hive.ql.plan.ColStatistics;
 
 /**
- * Enables statistics related computation on UDFs
+ * Enables statistics related computation on UDFs.
+ *
+ * <p>This interface provides two default implementations:
+ * <ul>
+ *   <li>{@link #estimate(List)} - clones the first argument's statistics (suitable for most UDFs)</li>
+ *   <li>{@link #estimate(List, long)} - calls estimate(List) and caps NDV at numRows</li>
+ * </ul>
+ *
+ * <p>UDFs that simply pass through statistics (like LOWER, UPPER) can use the defaults.
+ * UDFs that combine statistics (like IF, WHEN, COALESCE) should override {@link #estimate(List)}.

Review Comment:
   Since the respective methods have Javadoc the overview here is not strictly necessary and appears a bit repetitive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6359: HIVE-29368: More accurate pessimistic stats combining
URL: https://github.com/apache/hive/pull/6359


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4181627704

   @zabetak I think I have finally realized that while combining changes make little sense without computeNDVGroupingColumns() fixes, the actual problem in computeNDVGroupingColumns applies for non-combined columns too and we should likely fix it first. I will create a new story for that isolated fix


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030869220


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   @zabetak thank you for your feedback; I fully understand your concerns and the goal of reducing code changes in one PR.
   
   The main problem with keeping this code 100% intact is that **any** ColStatistics object coming into extractNDVGroupingColumns with an NDV of 0 and numNulls > 0 would be estimated with **1** GROUP BY row count; and this "1" estimate has proven to be the root cause of many severe under-estimations. 
   
   It makes this method logic directly correlated with the proposed changes to PessimisticStatCombiner
   
   Technically, the following two variants are feasible alternatives to the current PR code:
   1.
   `        if ((ndv > 0 && cs.getNumNulls() > 0) {
             ndv = StatsUtils.safeAdd(ndv, 1);
           }
   `
   ^^ This one adjusts NDV estimations except for columns with (potentially) unknown NDVs. Applying this change did not cause changes to any already modified .out files **of this branch**. I believe this is the safest possible change for this logic
   
   2. Completely removing the +1 adjustment for NULLs:
   `        ndvValues.add(cs.getCountDistint());
   `
   ^^ This code edit modifies only three files on the current branch
   
   If you firmly oppose any changes to **extractNDVGroupingColumns**, then we will likely need to get back to the drawing board on how to approach this whole problem in a different manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4111648307

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r2988776330


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -1578,7 +1578,7 @@ public static ColStatistics getColStatisticsFromExpression(HiveConf conf, Statis
             csList.add(cs);
           }
           if (csList.size() == engfd.getChildren().size()) {
-            Optional<ColStatistics> res = se.estimate(csList);
+            Optional<ColStatistics> res = se.estimate(csList, numRows);

Review Comment:
   The very first iteration had just his: https://github.com/apache/hive/pull/6359/changes/d90457f2c80ac2be43f1efe6bd1442ba9c508f39
   
   This felt a bit "too high level" to me, but thinking further, it indeed makes more sense to enforce here rather than on the estimator level, thank you @zabetak 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4053572012

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4130507448

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3012726545


##########
ql/src/test/results/clientpositive/llap/vectorized_stats.q.out:
##########
@@ -1207,13 +1207,13 @@ STAGE PLANS:
                       minReductionHashAggr: 0.99
                       mode: hash
                       outputColumnNames: _col0
-                      Statistics: Num rows: 1 Data size: 40 Basic stats: COMPLETE Column stats: COMPLETE
+                      Statistics: Num rows: 6144 Data size: 183480 Basic stats: COMPLETE Column stats: COMPLETE

Review Comment:
   @zabetak, this is a good example of the possible impact of the NDV==0 on statistics' estimates. Due to the issue https://issues.apache.org/jira/browse/HIVE-29534, the Metastore NDV (35) of the column table_onetimestamp.val1 is getting ignored, making the NDV of the underlying ColStatistics object 0. Later, extractNDVGroupingColumns() sets this 0 value to 1. The new estimate of 6144 is way worse than the metastore value of 35, but could be way better than the original "1" in certain queries. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4119404034

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [16 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [2.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r2988884907


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/PessimisticStatCombiner.java:
##########
@@ -41,9 +42,14 @@ public void add(ColStatistics stat) {
     if (stat.getAvgColLen() > result.getAvgColLen()) {
       result.setAvgColLen(stat.getAvgColLen());
     }
-    if (stat.getCountDistint() > result.getCountDistint()) {
-      result.setCountDistint(stat.getCountDistint());
+    // NDV=0 is "unknown" only if the stat is NOT a constant.
+    // Constants with NDV=0 (e.g., NULL) are "known zero", not unknown.
+    if ((result.getCountDistint() == 0 && !result.isConst()) || (stat.getCountDistint() == 0 && !stat.isConst())) {
+      result.setCountDistint(0);

Review Comment:
   @zabetak, this is the most complicated problem to solve, in my opinion. The following code: https://github.com/apache/hive/blob/931d4bb62b26de699240c816df439e00644e3dcb/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1634 is very explicit in assigning an NDV of 0 to NULL constants and 1 to non-NULL constants. At the same time, an NDV of "0" for a source column is typically used to indicate that the NDV for the column. is "unknown", which could matter a lot for large tables. Therefore, simply "summing" an NDV of 0 introduces even bigger mis-estimations
   
   To compensate for the "0 NDV" null constant, The following code: https://github.com/apache/hive/blob/931d4bb62b26de699240c816df439e00644e3dcb/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L2113 has a "+1" adjustment when figuring out NDVs of a GROUP BY. 
   
   I am thinking of modifying buildColStatForConstant() to treat NULL values as regular constants and see if we run into any significant side effects. If you have any additional thoughts on the subject, I would greatly appreciate knowing those. Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4145799668

   @zabetak @okumin 
   I was unable to find a way to distinguish between "const NULL" ColStatistics objects and objects that truly have an "unknown NDV" without changing the ColStatistics class **or** refactoring the StatEstimator interface, and making PessimisticStatCombiner aware of the total # of rows from the parent stats. 
   
   I believe that the latest changes for StatEstimator are minimally invasive and fully backwards-compatible. If this still feels "too intrusive", I'd appreciate some hints on possible alternatives 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3016466506


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/StatEstimator.java:
##########
@@ -39,5 +40,19 @@ public interface StatEstimator {
    * @param argStats the statistics for every argument of the UDF
    * @return {@link ColStatistics} estimate for the actual UDF.
    */
-  public Optional<ColStatistics> estimate(List<ColStatistics> argStats);
+  default Optional<ColStatistics> estimate(List<ColStatistics> argStats) {
+    throw new UnsupportedOperationException("This estimator requires parentStats");
+  }

Review Comment:
   Do we need to change this to default?



##########
ql/src/test/results/clientpositive/llap/vectorized_stats.q.out:
##########
@@ -1207,13 +1207,13 @@ STAGE PLANS:
                       minReductionHashAggr: 0.99
                       mode: hash
                       outputColumnNames: _col0
-                      Statistics: Num rows: 1 Data size: 40 Basic stats: COMPLETE Column stats: COMPLETE
+                      Statistics: Num rows: 6144 Data size: 183480 Basic stats: COMPLETE Column stats: COMPLETE

Review Comment:
   Thanks for the explanation. Since the new estimate is not always better maybe we should rediscuss the change in extractNDVGroupingColumns or maybe address HIVE-29534 first.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/PessimisticStatCombiner.java:
##########
@@ -21,16 +21,26 @@
 import java.util.Optional;
 
 import org.apache.hadoop.hive.ql.plan.ColStatistics;
+import org.apache.hadoop.hive.ql.stats.StatsUtils;
 
 /**
  * Combines {@link ColStatistics} objects to provide the most pessimistic estimate.
  */
 public class PessimisticStatCombiner {
 
+  private final long numRows;
   private boolean inited;
+  private boolean hasUnknownNDV;
   private ColStatistics result;
 
+  public PessimisticStatCombiner(long numRows) {
+    this.numRows = numRows;
+  }
+
   public void add(ColStatistics stat) {
+    // NDV==0 means unknown, unless it's a NULL constant (numNulls == numRows)
+    hasUnknownNDV = hasUnknownNDV || (stat.getCountDistint() == 0 && stat.getNumNulls() != numRows);
+

Review Comment:
   Note that if in the CASE statement you have one branch with unknown NDV and another that is the NULL constant the result of the combiner will appear to other consumers as a NULL constant according to the criteria specified here. Probably an edge case but still wanted to highlight that its hard to cover everything from the moment that NDV = 0 is a valid value.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   Another reason why I suggest to defer this on another patch is cause I am not fully convinced if we should actually do a +1 on the NDV for nulls. If we continue on the assumption that is used elsewhere that null is not a value and thus does not contribute to NDV then we shouldn't modify the NDV; possibly we should set the `numNulls` to 1.



##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   It seems that this change led to more .q.out changes and addresses an NDV issue on a slightly different place.  Would it be possible to defer this to another patch/Jira so that we converge and merge this PR sooner. I understand the intention of the change but I am a bit reluctant to modify multiple things as part of the same change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030869220


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   @zabetak thank you for your feedback; I fully understand your concerns and the goal of reducing code changes in one PR.
   
   The main problem with keeping this code 100% intact is that **any** ColStatistics object coming into extractNDVGroupingColumns with an NDV of 0 and numNulls > 0 would be estimated with **1** GROUP BY row count; and this "1" estimate has proven to be the root cause of many severe under-estimations. 
   
   It makes this method logic directly correlated with the proposed changes to PessimisticStatCombiner
   
   Technically, the following two variants are feasible alternatives to the current PR code:
   1.
   `        if ((ndv > 0 && cs.getNumNulls() > 0) {
             ndv = StatsUtils.safeAdd(ndv, 1);
           }`
   
   ^^ This one adjusts NDV estimations except for columns with (potentially) unknown NDVs. Applying this change did not cause changes to any already modified .out files **of this branch**. I believe this is the safest possible change for this logic
   
   2. Completely removing the +1 adjustment for NULLs:
   `ndvValues.add(cs.getCountDistint());`
   ^^ This code edit modifies only three files on the current branch
   
   If you firmly oppose any changes to **extractNDVGroupingColumns**, then we will likely need to get back to the drawing board on how to approach this whole problem in a different manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4157427860

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030869220


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   @zabetak thank you for your feedback; I fully understand your concerns and the goal of reducing code changes in one PR.
   
   The main problem with keeping this code 100% intact is that **any** ColStatistics object coming into extractNDVGroupingColumns with an NDV of 0 and numNulls > 0 would be estimated with **1** GROUP BY row count; and this "1" estimate has proven to be the root cause of many severe under-estimations. 
   
   It makes this method logic directly correlated with the proposed changes to PessimisticStatCombiner
   
   Technically, the following two variants are feasible alternatives to the current PR code:
   1.
   `        if ((ndv > 0 && cs.getNumNulls() > 0) {
             ndv = StatsUtils.safeAdd(ndv, 1);
           }`
   
   ^^ This one adjusts NDV estimations except for columns with (potentially) unknown NDVs. Applying this change did not cause changes to any already modified .out files **of this branch**. I believe this is the safest possible change for this logic
   
   2. Completely removing the +1 adjustment for NULLs:
   `
           ndvValues.add(cs.getCountDistint());
   `
   ^^ This code edit modifies only three files on the current branch
   
   If you firmly oppose any changes to **extractNDVGroupingColumns**, then we will likely need to get back to the drawing board on how to approach this whole problem in a different manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4122522348

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r2994697977


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/PessimisticStatCombiner.java:
##########
@@ -41,9 +42,14 @@ public void add(ColStatistics stat) {
     if (stat.getAvgColLen() > result.getAvgColLen()) {
       result.setAvgColLen(stat.getAvgColLen());
     }
-    if (stat.getCountDistint() > result.getCountDistint()) {
-      result.setCountDistint(stat.getCountDistint());
+    // NDV=0 is "unknown" only if the stat is NOT a constant.
+    // Constants with NDV=0 (e.g., NULL) are "known zero", not unknown.
+    if ((result.getCountDistint() == 0 && !result.isConst()) || (stat.getCountDistint() == 0 && !stat.isConst())) {
+      result.setCountDistint(0);

Review Comment:
   I checked a bit the code and it appears that nulls should not add up to NDV. For example:
   ```sql
   SELECT CASE WHEN cond=1 THEN NULL WHEN cond=2 THEN 'A' ELSE 'B' END x FROM t
   ```
   NDV(x) = 2
   NUM_NULLS(x) = |t|
   
   so it might be better to keep things as are and don't modify `buildColStatForConstant`.
   
   In fact, everything would be simpler if we could use -1 for NDV to declare unknown as it happens for the other stats. This is probably a bigger change to digest so let's not go into this direction for now.
   
   I understand the risk that summing NDV with zero can lead to bad estimations but it is still an improvement against the current state of the code (at least it does not worsen the situation). We could also be a bit more defensive and say that if numNulls >= 0 then NDV is valid and safe to use (not unknown). Still if you consider this to be very risky then I am OK to revert to your original proposal of adding a new boolean field in `ColStatistics` object for marking constants. Leaving the final choice to you :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4580455972

   @zabetak, my sincerest apologies for not pinging you earlier; moving this one into a draft now. When you get a chance, could you please take a look at https://github.com/apache/hive/pull/6505 - an alternative solution aimed at replacing this PR completely?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: Could this be the truly accurate pessimistic stats combining? [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6359:
URL: https://github.com/apache/hive/pull/6359#issuecomment-4102250406

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6359&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6359&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6359&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6359)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030869220


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   @zabetak thank you for your feedback; I fully understand your concerns and the goal of reducing code changes in one PR.
   
   The main problem with keeping this code 100% intact is that **any** ColStatistics object coming into extractNDVGroupingColumns with an NDV of 0 and numNulls > 0 would be estimated with **1** GROUP BY row count; and this "1" estimate has proven to be the root cause of many severe under-estimations. 
   
   It makes this method logic directly correlated with the proposed changes to PessimisticStatCombiner
   
   Technically, the following two variants are feasible alternatives to the current PR code:
   1.
   `
           if ((ndv > 0 && cs.getNumNulls() > 0) {
             ndv = StatsUtils.safeAdd(ndv, 1);
           }
   `
   ^^ This one adjusts NDV estimations except for columns with (potentially) unknown NDVs. Applying this change did not cause changes to any already modified .out files **of this branch**. I believe this is the safest possible change for this logic
   
   2. Completely removing the +1 adjustment for NULLs:
   `
           ndvValues.add(cs.getCountDistint());
   `
   ^^ This code edit modifies only three files on the current branch
   
   If you firmly oppose any changes to **extractNDVGroupingColumns**, then we will likely need to get back to the drawing board on how to approach this whole problem in a different manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030869220


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java:
##########
@@ -2109,7 +2111,10 @@ private static List<Long> extractNDVGroupingColumns(List<ColStatistics> colStats
     for (ColStatistics cs : colStats) {
       if (cs != null) {
         long ndv = cs.getCountDistint();
-        if (cs.getNumNulls() > 0) {
+        // NDV needs to be adjusted if a column has a known NDV along with NULL values
+        // or if a column happens to be "const NULL"
+        if ((ndv > 0 && cs.getNumNulls() > 0) ||
+            (ndv == 0 && !cs.isEstimated() && cs.getNumNulls() == parentStats.getNumRows())) {

Review Comment:
   @zabetak thank you for your feedback; I fully understand your concerns and the goal of reducing code changes in one PR.
   
   The main problem with keeping this code 100% intact is that **any** ColStatistics object coming into extractNDVGroupingColumns with an NDV of 0 and numNulls > 0 would be estimated with **1** GROUP BY row count; and this "1" estimate has proven to be the root cause of many severe under-estimations. 
   
   It makes this method logic directly correlated with the proposed changes to PessimisticStatCombiner
   
   Technically, the following two alternatives are feasible alternatives to the current PR code:
   1.
   `        if ((ndv > 0 && cs.getNumNulls() > 0) {
             ndv = StatsUtils.safeAdd(ndv, 1);
           }
   `
   ^^ This one adjusts NDV estimations except for columns with (potentially) unknown NDVs. Applying this change did not cause changes to any already modified .out files **of this branch**. I believe this is the safest possible change for this logic
   
   2. Completely removing the +1 adjustment for NULLs:
   ` ndvValues.add(cs.getCountDistint());`
   ^^ This code edit modifies only three files on the current branch
   
   If you firmly oppose any changes to **extractNDVGroupingColumns**, then we will likely need to get back to the drawing board on how to approach this whole problem in a different manner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3030882052


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/StatEstimator.java:
##########
@@ -39,5 +40,19 @@ public interface StatEstimator {
    * @param argStats the statistics for every argument of the UDF
    * @return {@link ColStatistics} estimate for the actual UDF.
    */
-  public Optional<ColStatistics> estimate(List<ColStatistics> argStats);
+  default Optional<ColStatistics> estimate(List<ColStatistics> argStats) {
+    throw new UnsupportedOperationException("This estimator requires parentStats");
+  }

Review Comment:
   This change eliminates the need to implement estimate(args) for impacted estimators while allowing other estimators that do not need numRows to keep their original estimate(args) implementation, without having to implement estimate(args, numRows). I.e., it felt "cleaner" and allowed me to reduce the # of Java files that required edits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29368: More accurate pessimistic stats combining [hive]

Posted by "konstantinb (via GitHub)" <gi...@apache.org>.
konstantinb commented on code in PR #6359:
URL: https://github.com/apache/hive/pull/6359#discussion_r3028888736


##########
ql/src/java/org/apache/hadoop/hive/ql/stats/estimator/PessimisticStatCombiner.java:
##########
@@ -21,16 +21,26 @@
 import java.util.Optional;
 
 import org.apache.hadoop.hive.ql.plan.ColStatistics;
+import org.apache.hadoop.hive.ql.stats.StatsUtils;
 
 /**
  * Combines {@link ColStatistics} objects to provide the most pessimistic estimate.
  */
 public class PessimisticStatCombiner {
 
+  private final long numRows;
   private boolean inited;
+  private boolean hasUnknownNDV;
   private ColStatistics result;
 
+  public PessimisticStatCombiner(long numRows) {
+    this.numRows = numRows;
+  }
+
   public void add(ColStatistics stat) {
+    // NDV==0 means unknown, unless it's a NULL constant (numNulls == numRows)
+    hasUnknownNDV = hasUnknownNDV || (stat.getCountDistint() == 0 && stat.getNumNulls() != numRows);
+

Review Comment:
   I have struggled with this too; technically, in the code before these changes, this makes the combined stats for the column come out with numNulls == numRows. The "estimated" flag could be used to decide how much trust a consumer should put to such statistics entries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org