You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "soumyakanti3578 (via GitHub)" <gi...@apache.org> on 2025/08/06 01:33:06 UTC

[PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

soumyakanti3578 opened a new pull request, #6009:
URL: https://github.com/apache/hive/pull/6009

   …LAST_VALUE
   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
     2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
     3. Ensure you have added or run the appropriate tests for your PR: 
     4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX:  Your PR title ...'.
     5. Be sure to keep the PR description updated to reflect all changes.
     6. Please write your PR title to summarize what this PR proposes.
     7. If possible, provide a concise example to reproduce the issue for a faster review.
   
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Pass `respectNulls` to the `first_value` and `last_value` VectorPTFEvaluator classes and update `evaluateGroupBatch` methods to support `IGNORE NULLS`
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Explained in detail: https://issues.apache.org/jira/browse/HIVE-29122
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   No
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   ```
   mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver -Dtest.output.overwrite=true -Dqfile="vectorized_first_last_value_ignore_nulls_double.q,vectorized_first_last_value_ignore_nulls_int.q,vectorized_first_last_value_ignore_nulls_decimal.q"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz merged PR #6009:
URL: https://github.com/apache/hive/pull/6009


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3160129849

   @soumyakanti3578 
   How cases handled when the rows in the window does not fit into one batch?
   * Case of `first_value respect nulls` is easy because we take the first row in the first batch.
   * Case of `last_value respect nulls` is more challenging because we need the value from the last batch. However the last row in the window is not always the last row in the batch. I mean it is not `batch.size - 1`. Is this true?
   * Case of `first_value ignore nulls`. What if all the values are null in the first batch? We need to check the follow up batches.
   * Case of `last_value ignore nulls`. As you already implemented in this patch iterate back from `batch.size - 1` until we find a non-null value. This is good when we have one batch but it is more challenging when we have follow-up batches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3164409020

   @soumyakanti3578 
   Thanks for the detailed explanation. Overall the patch seems to be ok to me.
   
   One note for the testing part: IMHO it worth adding test cases for both:
   * batch size is big enough to load all the records into one batch from the table
   * or batch size is too small and the windows are loaded into more than one batch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "soumyakanti3578 (via GitHub)" <gi...@apache.org>.
soumyakanti3578 commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3162054831

   @kasakrisz 
   I debugged a bit to understand the workflow. To simulate the scenario where rows with the same key doesn't fit into 1 single batch, we can run the tests with `SET hive.vectorized.testing.reducer.batch.size=2;` With this I have noticed that `VectorPTFGroupBatches.bufferedBatches` contains batches whose size is at most 2. 
   
   For the data in `vectorized_first_last_value_ignore_nulls_int.q`, for key 1 and 2, I have seen batches like:
   ```
   Batch 1:   [[1, null], [1, null]]
   Batch 2:   [1, null]
   Batch 3:   [[2, null], [2, 20]]
   Batch 4:   [2, null]
   ```
   Let's look at `last_value ignore nulls` computation.
   First we will compute last value for Range with key = 1, which consists of Batch 1 and 2. A new evaluator will be created for `VectorPTFEvaluatorLongLastValue`, which we will keep reusing.
   
   First, we will call `VectorPTFEvaluatorLongLastValue.evaluateGroupBatch(batch 1)` will be called, and we will have to check both vector elements for non null values. Since they are null, we will have `isGroupResultNull = true` and `lastValue = null`.
   
   Then, we will call `VectorPTFEvaluatorLongLastValue.evaluateGroupBatch(batch 2)` on the same evaluator object, but the  value of `isGroupResultNull` and `lastValue` doesn't change.
   
   Next we will call `VectorPTFEvaluatorLongLastValue.evaluateGroupBatch(batch 3)` on the same evaluator, and this time, the values will be `isGroupResultNull = false` and `lastValue = 20`.
   
   And then, we will call `VectorPTFEvaluatorLongLastValue.evaluateGroupBatch(batch 4)` on the same evaluator, and the values will be `isGroupResultNull = true` and `lastValue = 20`. Note that `lastValue` remains unchanged.
   
   Now if we didn't have other rows in the table, we would get out of the `while` loop in `VectorPTFGroupBatches.BlockIterator.run` and call `getEvaluatorResult(evaluator)` as seen below:
   ```
       Object run(VectorPTFEvaluatorBase evaluator) throws HiveException {
         runEvaluator(evaluator, range);
   
         while (hasNext()) {
           nextBlock();
           runEvaluator(evaluator, range);
         }
   
         return getEvaluatorResult(evaluator);
       }
   
       Object getEvaluatorResult(VectorPTFEvaluatorBase evaluator) {
         return evaluator.isGroupResultNull() ? null : evaluator.getGroupResult();
       }
   ```
   
   `VectorPTFEvaluatorLongLastValue.isGroupResultNull()` was updated in this PR as:
   ```
     public boolean isGroupResultNull() {
       return isGroupResultNull && doesRespectNulls();
     }
   ```
   This returns `false` even though `isGroupResultNull = true` at the end because `doesRespectNulls()` returns false. So finally `evaluator.getGroupResult()` is called, which returns `lastValue = 20`.
   
   If we were running `last value respect nulls`, it would have returned `null` because `isGroupResultNull()` would have returned `true`.
   
   Similarly `first value ignore nulls` can be explained too. One important difference is that `VectorPTFEvaluatorLongFirstValue` has the boolean variable `haveFirstValue`. Once we find a non null value, this variable is set to true and for all batches after that we directly write the value into the output vector.
   
   Hopefully this makes sense, but please let me know if I am missing something. If needed, I can add `SET hive.vectorized.testing.reducer.batch.size=2;` to all three test files too - they are all passing on my local machine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3164161482

   > And then, we will call VectorPTFEvaluatorLongLastValue.evaluateGroupBatch(batch 4) on the same evaluator, and the values will be isGroupResultNull = true and lastValue = 20. Note that lastValue remains unchanged.
   
   shouldn't the `isGroupResultNull` stay false? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #6009:
URL: https://github.com/apache/hive/pull/6009#discussion_r2260492269


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/VectorPTFDesc.java:
##########
@@ -321,6 +323,7 @@ public static VectorPTFEvaluatorBase getEvaluator(SupportedFunctionType function
       default:
         throw new RuntimeException("Unexpected column vector type " + columnVectorType + " for " + functionType);
       }
+      evaluator.setRespectNulls(respectNulls);

Review Comment:
   Could you please move this call to the end of the method before returning the `evaluator` instance? In that case this would be called only once int the entire method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "soumyakanti3578 (via GitHub)" <gi...@apache.org>.
soumyakanti3578 commented on code in PR #6009:
URL: https://github.com/apache/hive/pull/6009#discussion_r2261558586


##########
ql/src/java/org/apache/hadoop/hive/ql/plan/VectorPTFDesc.java:
##########
@@ -321,6 +323,7 @@ public static VectorPTFEvaluatorBase getEvaluator(SupportedFunctionType function
       default:
         throw new RuntimeException("Unexpected column vector type " + columnVectorType + " for " + functionType);
       }
+      evaluator.setRespectNulls(respectNulls);

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "soumyakanti3578 (via GitHub)" <gi...@apache.org>.
soumyakanti3578 commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3166006868

   @deniskuzZ 
   Yes, but it didn't affect the results. But in the latest commit I have updated the initial value to `isGroupResultNull = lastValue == null;` which should take care of the inconsistency :) 
   
   @kasakrisz 
   I have added more tests in all three files with `set hive.vectorized.testing.reducer.batch.size=2;`
   
   Please review again! Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3166116711

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6009) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6009&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6009&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6009&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6009&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [11.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6009&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6009)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3166667347

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6009) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6009&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6009&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6009&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6009&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [11.8% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6009&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6009)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-29122: Vectorization - Support IGNORE NULLS for FIRST_VALUE and … [hive]

Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6009:
URL: https://github.com/apache/hive/pull/6009#issuecomment-3157216552

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6009) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [6 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6009&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6009&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6009&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6009&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [15.1% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6009&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6009)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org