You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ggangadharan (via GitHub)" <gi...@apache.org> on 2026/10/08 17:58:40 UTC
[PR] HIVE-30018: PTF: Vectorize ntile function [hive]
ggangadharan opened a new pull request, #6861:
URL: https://github.com/apache/hive/pull/6861
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Adds a vectorized evaluator for `ntile(n)`, so a PTF operator that uses ntile no longer forces its reducer into row mode.
- **`VectorPTFEvaluatorNTile`** (new): a group-aggregated streaming evaluator, like `cume_dist` / `percent_rank`. The partition is buffered so its size is known, then bucket numbers are derived on the fly while the batches are replayed. State is O(1) per partition (no per-row array), and a batch that falls entirely within one bucket is emitted as a repeating vector. The bucket layout matches `GenericUDAFNTile`: the first `partitionSize % n` buckets get one extra row.
- **`VectorPTFDesc`**: registers `NTILE` and adds it to `COLUMN_AGNOSTIC_FUNCTIONS`, since its argument is a constant rather than an input column.
- **`Vectorizer`**: vectorizes ntile only when the number of buckets is a positive int constant. Row mode also accepts a non-c
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Part of HIVE-24872 (PTF vectorization coverage). Until now, any PTF containing ntile ran its whole reducer in row mode:
```
notVectorizedReason: PTF operator: ntile not in supported functions [avg, count, cume_dist, dense_rank, first_value, lag, last_value, lead, max, min, percent_rank, rank, row_number, sum]
```
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducible example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No change in query results. Queries using `ntile(<constant>)` can now run with a vectorized reducer, which is visible in EXPLAIN:
```sql
explain vectorization detail
select name, rowindex, mynumber,
ntile(4) over (partition by name order by mynumber) as nt
from vector_ptf_ntile_int;
```
- Before: `Reducer 2` shows `Execution mode: llap` and the `notVectorizedReason` above.
- After: `Reducer 2` shows `Execution mode: vectorized, llap` and `evaluatorClasses: [VectorPTFEvaluatorNTile]`.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
- **New unit test `TestVectorPTFEvaluatorNTile`** (17 tests): compared against the `GenericUDAFNTile` algorithm for partition sizes 0-25 x 1-10 buckets x several batch sizes; repeating and non-repeating output; reuse across partitions; a ~100M-row partition; error cases.
- **New qtest `vector_ptf_ntile.q`**: every query runs with PTF vectorization off and then on, with identical results. Covers partitioned, unpartitioned and `over ()` windows, more buckets than rows, a NULL partition, NULL order keys, ntile mixed with `rank` / `cume_dist` / `sum` in one PTF operator, and the non-constant fallback. The reducer batch size is 2 so bucket boundaries fall across batches.
- **Updated q files**, with no result changes: `vector_windowing.q.out` (ntile is now in the supported list; that query is now blocked by `stddev_pop`) and `sketches_materialized_view_ntile.q.out` (one reducer is now vectorized).
```
mvn test -pl ql -Dtest=TestVectorPTFEvaluatorNTile,TestVectorPTFGroupBatches
mvn test -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver \
-Dqfile=vector_ptf_ntile.q,vector_windowing.q,windowing.q,windowing_ntile.q,ptf_decimal.q,cbo_rp_windowing_2.q,lineage_ptf.q,sketches_materialized_view_ntile.q
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ramitg254 (via GitHub)" <gi...@apache.org>.
ramitg254 commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4255999719
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
change makes sense, I know I am asking too much but can you please remove workaround added in these files:
ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist.q
ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist_partition_by.q
in this change https://github.com/apache/hive/pull/6703/changes#diff-e03a7feea22c2de54e39d2a81840303b5fd18d4bdfdf15be9af2eab68984f5cb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4269373439
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -3011,6 +3012,13 @@ private boolean validatePTFOperator(PTFOperator op, VectorizationContext vContex
throw new RuntimeException("Unexpected window type " + windowFrameDef.getWindowType());
}
+ // Row mode also accepts a non-constant ntile argument (read from the first row of the partition).
+ if (supportedFunctionType == SupportedFunctionType.NTILE &&
+ !isPositiveIntConstant(singleExprNodeDesc)) {
+ setOperatorIssue(functionName + " only a positive integer constant number of buckets is supported");
+ return false;
Review Comment:
`isPositiveIntConstant` only accepts constants whose runtime value is an `Integer`. In Hive expression trees, numeric constants can legitimately be represented as other numeric types (e.g., `Long`, `Short`, `Byte`, or writable equivalents depending on constant folding / type inference), and those would be incorrectly rejected and force row-mode even though the evaluator can handle any positive value within int range. Consider accepting any `Number` (or using the constant's `TypeInfo` + numeric conversion) and validating `> 0` and `<= Integer.MAX_VALUE`.
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -5042,6 +5050,14 @@ private static ExprNodeDesc[] getOrderExprNodeDescs(List<OrderExpressionDef> ord
return exprNodeDescs;
}
+ private static boolean isPositiveIntConstant(ExprNodeDesc exprNodeDesc) {
+ if (!(exprNodeDesc instanceof ExprNodeConstantDesc)) {
+ return false;
+ }
+ Object value = ((ExprNodeConstantDesc) exprNodeDesc).getValue();
+ return value instanceof Integer && (Integer) value > 0;
Review Comment:
`isPositiveIntConstant` only accepts constants whose runtime value is an `Integer`. In Hive expression trees, numeric constants can legitimately be represented as other numeric types (e.g., `Long`, `Short`, `Byte`, or writable equivalents depending on constant folding / type inference), and those would be incorrectly rejected and force row-mode even though the evaluator can handle any positive value within int range. Consider accepting any `Number` (or using the constant's `TypeInfo` + numeric conversion) and validating `> 0` and `<= Integer.MAX_VALUE`.
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -3011,6 +3012,13 @@ private boolean validatePTFOperator(PTFOperator op, VectorizationContext vContex
throw new RuntimeException("Unexpected window type " + windowFrameDef.getWindowType());
}
+ // Row mode also accepts a non-constant ntile argument (read from the first row of the partition).
+ if (supportedFunctionType == SupportedFunctionType.NTILE &&
+ !isPositiveIntConstant(singleExprNodeDesc)) {
+ setOperatorIssue(functionName + " only a positive integer constant number of buckets is supported");
Review Comment:
This new validation message is grammatically awkward and a bit hard to parse. Consider rephrasing to something like: `functionName + \" only supports a positive integer constant number of buckets\"` or `functionName + \": number of buckets must be a positive integer constant\"` to make EXPLAIN output clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6151070360
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270424899
##########
ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist.q:
##########
@@ -12,15 +12,11 @@ insert into table sketch_input values
select id,cume_dist() over (order by id) from sketch_input;
--- Vectorized PTF buffered path does not support complex-type passthrough columns.
-set hive.vectorized.execution.ptf.enabled=false;
Review Comment:
Yes, the same scratch-column bug. HIVE-25124 added the workaround for it. That's now explained in the "Related fix" section of the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4269304308
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -3011,6 +3012,13 @@ private boolean validatePTFOperator(PTFOperator op, VectorizationContext vContex
throw new RuntimeException("Unexpected window type " + windowFrameDef.getWindowType());
}
+ // Row mode also accepts a non-constant ntile argument (read from the first row of the partition).
+ if (supportedFunctionType == SupportedFunctionType.NTILE &&
Review Comment:
please create a follow-up ticket for parity with non-vectorized mode
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+
+ public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
+ VectorExpression numBucketsExpression, int outputColumnNum) {
+ super(windowFrameDef, outputColumnNum);
+ if (!(numBucketsExpression instanceof ConstantVectorExpression)) {
+ throw new RuntimeException(
+ "NTILE expects a constant integer expression for the number of buckets, got: "
+ + (numBucketsExpression == null ? "null"
+ : numBucketsExpression.getClass().getName()));
+ }
+ long value = ((ConstantVectorExpression) numBucketsExpression).getLongValue();
+ if (value <= 0 || value > Integer.MAX_VALUE) {
+ throw new RuntimeException(
+ "NTILE requires a positive integer number of buckets, got: " + value);
+ }
+ this.numBuckets = (int) value;
Review Comment:
what about simply using `long numBuckets` field?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+
+ public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
+ VectorExpression numBucketsExpression, int outputColumnNum) {
+ super(windowFrameDef, outputColumnNum);
+ if (!(numBucketsExpression instanceof ConstantVectorExpression)) {
Review Comment:
an extra cast can be removed by:
```
if (!(numBucketsExpression instanceof ConstantVectorExpression constantExpr)) {
```
##########
ql/src/test/results/clientpositive/llap/vector_windowing.q.out:
##########
@@ -3250,7 +3250,7 @@ STAGE PLANS:
Reduce Vectorization:
enabled: true
enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez] IS true
- notVectorizedReason: PTF operator: ntile not in supported functions [avg, count, cume_dist, dense_rank, first_value, lag, last_value, lead, max, min, percent_rank, rank, row_number, sum]
+ notVectorizedReason: PTF operator: stddev_pop not in supported functions [avg, count, cume_dist, dense_rank, first_value, lag, last_value, lead, max, min, ntile, percent_rank, rank, row_number, sum]
Review Comment:
consider creating follow-up ticket for `stddev_pop` vectorization
##########
ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist.q:
##########
@@ -12,15 +12,11 @@ insert into table sketch_input values
select id,cume_dist() over (order by id) from sketch_input;
--- Vectorized PTF buffered path does not support complex-type passthrough columns.
-set hive.vectorized.execution.ptf.enabled=false;
Review Comment:
this might also be related to the scratch col related change, isn't it? if so, let's have same clarity about it in the PR desription
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -474,13 +476,31 @@ private void initBufferedColumns() {
bufferedTypeInfos[bufferedMapIndex] = outputTypeInfos[streamingEvaluatorNum];
}
- for (int i = 0; i < scratchColumnPositions.length; i++) {
+ for (int i = 0; i < bufferedScratchIndices.length; i++) {
+ final int scratchIndex = bufferedScratchIndices[i];
final int bufferedMapIndex = orderColumnCount + nonKeyInputColumnCount + streamingEvaluatorCount + i;
- bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[i];
- bufferedTypeInfos[bufferedMapIndex] = TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[i]);
+ bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[scratchIndex];
+ bufferedTypeInfos[bufferedMapIndex] =
+ TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[scratchIndex]);
}
}
+ /**
+ * Returns the indices (into scratchColumnPositions) of the scratch columns written by evaluator
+ * input expressions. Those are evaluated before a batch is buffered, so their results must be
+ * buffered too. The other scratch columns belong to child operators, which recompute them after
+ * the batch is forwarded; buffering them is wasted work and fails for complex types.
+ */
+ private int[] getEvaluatorInputScratchIndices() {
Review Comment:
is the change related to introducing ntile function? I feel it's unrelated, please clarify it in the PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270410278
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -3011,6 +3012,13 @@ private boolean validatePTFOperator(PTFOperator op, VectorizationContext vContex
throw new RuntimeException("Unexpected window type " + windowFrameDef.getWindowType());
}
+ // Row mode also accepts a non-constant ntile argument (read from the first row of the partition).
+ if (supportedFunctionType == SupportedFunctionType.NTILE &&
Review Comment:
Created [HIVE-30162](https://issues.apache.org/jira/browse/HIVE-30162) as a subtask of HIVE-24872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270448722
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -5042,6 +5050,14 @@ private static ExprNodeDesc[] getOrderExprNodeDescs(List<OrderExpressionDef> ord
return exprNodeDescs;
}
+ private static boolean isPositiveIntConstant(ExprNodeDesc exprNodeDesc) {
+ if (!(exprNodeDesc instanceof ExprNodeConstantDesc)) {
+ return false;
+ }
+ Object value = ((ExprNodeConstantDesc) exprNodeDesc).getValue();
+ return value instanceof Integer && (Integer) value > 0;
Review Comment:
Only int can reach this point: GenericUDAFNTile.getEvaluator rejects any other argument type during semantic analysis ("Number of tiles must be an int expression"). I checked ntile(4Y), ntile(4S) and ntile(4L) (all rejected before vectorization) and ntile(2 + 2) (folds to an int constant and vectorizes). Keeping the Integer check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4247431160
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
Without this change, first commit failed CI on sketches_rewrite_ntile.q and sketches_rewrite_ntile_partition_by.q with below ERROR -
` org.apache.hadoop.hive.ql.exec.tez.TezRuntimeException: Vertex failed, vertexName=Reducer 3, vertexId=vertex_1791496978238_0001_162_03, diagnostics=[Task failed, taskId=task_1791496978238_0001_162_03_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1791496978238_0001_162_03_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:342)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:86)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:72)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:42)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:112)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:411)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:259)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:292)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:294)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:519)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:402)
... 18 more
Caused by: java.lang.RuntimeException: No complex type support: LIST
at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.copyNonSelectedColumnVector(VectorizedBatchUtil.java:802)
at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.copyNonSelectedColumnVector(VectorizedBatchUtil.java:644)
at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.copyBufferedColumns(VectorPTFGroupBatches.java:1265)
at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.bufferGroupBatch(VectorPTFGroupBatches.java:545)
at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.process(VectorPTFOperator.java:421)
at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:920)
at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:159)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)
... 19 more`
This fixes sketches_rewrite_ntile.q and sketches_rewrite_ntile_partition_by.q, which failed in CI on the first commit with No complex type support: LIST.
initBufferedColumns buffered every scratch column of the reducer, including the array<double> that the Select after the PTF computes for ds_kll_cdf(...)[0], and VectorizedBatchUtil.copyNonSelectedColumnVector can't copy lists. Since HIVE-29696, only evaluator input expressions are evaluated before buffering, so those are the only scratch columns a buffered batch needs. The rest belong to child operators, which recompute them after the batch is forwarded (prepareBufferedBatchForForwarding resets the overflow batch first). It isn't ntile-specific: the same query with cume_dist() instead of ntile fails the same way. It only showed up now because vectorizing ntile sends those queries through the buffered path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ramitg254 (via GitHub)" <gi...@apache.org>.
ramitg254 commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4256023652
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -474,13 +475,31 @@ private void initBufferedColumns() {
bufferedTypeInfos[bufferedMapIndex] = outputTypeInfos[streamingEvaluatorNum];
}
- for (int i = 0; i < scratchColumnPositions.length; i++) {
+ for (int i = 0; i < bufferedScratchIndices.length; i++) {
+ final int scratchIndex = bufferedScratchIndices[i];
final int bufferedMapIndex = orderColumnCount + nonKeyInputColumnCount + streamingEvaluatorCount + i;
- bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[i];
- bufferedTypeInfos[bufferedMapIndex] = TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[i]);
+ bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[scratchIndex];
+ bufferedTypeInfos[bufferedMapIndex] =
+ TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[scratchIndex]);
}
}
+ /**
+ * Returns the indices (into scratchColumnPositions) of the scratch columns written by evaluator
+ * input expressions. Those are evaluated before a batch is buffered, so their results must be
+ * buffered too. The other scratch columns belong to child operators, which recompute them after
+ * the batch is forwarded; buffering them is wasted work and fails for complex types.
+ */
+ private int[] getEvaluatorInputScratchIndices() {
+ final int[] inputExprColumns = Arrays.stream(evaluators)
+ .filter(evaluator -> evaluator.inputVecExpr != null)
+ .mapToInt(evaluator -> evaluator.inputVecExpr.getOutputColumnNum())
+ .toArray();
+ return IntStream.range(0, scratchColumnPositions.length)
Review Comment:
can we use a map for inputExprColumns then perform filtering using it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4247397441
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,385 @@
+/*
Review Comment:
Did you mean this complete JUNIT testcase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6174032081
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6069492001
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [17 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [0.9% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #6861:
URL: https://github.com/apache/hive/pull/6861
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6133319759
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4247397441
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,385 @@
+/*
Review Comment:
Did you mean this complete JUNIT testcase? is it due to other vectorized PTF functions (cume_dist, percent_rank, rank, …) has a unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6160144186
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6078631092
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [22 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [0.9% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270441611
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##########
@@ -3011,6 +3012,13 @@ private boolean validatePTFOperator(PTFOperator op, VectorizationContext vContex
throw new RuntimeException("Unexpected window type " + windowFrameDef.getWindowType());
}
+ // Row mode also accepts a non-constant ntile argument (read from the first row of the partition).
+ if (supportedFunctionType == SupportedFunctionType.NTILE &&
+ !isPositiveIntConstant(singleExprNodeDesc)) {
+ setOperatorIssue(functionName + " only a positive integer constant number of buckets is supported");
Review Comment:
Done, reworded to "ntile only supports a positive integer constant number of buckets".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4248244735
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,385 @@
+/*
Review Comment:
In this context nothing in this change needs a JUnit test. Hence added one more test in the q file and removed the complete JUNIT file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270420550
##########
ql/src/test/results/clientpositive/llap/vector_windowing.q.out:
##########
@@ -3250,7 +3250,7 @@ STAGE PLANS:
Reduce Vectorization:
enabled: true
enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez] IS true
- notVectorizedReason: PTF operator: ntile not in supported functions [avg, count, cume_dist, dense_rank, first_value, lag, last_value, lead, max, min, percent_rank, rank, row_number, sum]
+ notVectorizedReason: PTF operator: stddev_pop not in supported functions [avg, count, cume_dist, dense_rank, first_value, lag, last_value, lead, max, min, ntile, percent_rank, rank, row_number, sum]
Review Comment:
Created [HIVE-30163](https://issues.apache.org/jira/browse/HIVE-30163) as a subtask of HIVE-24872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6183571478
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4248230885
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+ private int rowsLeftInPartition;
+
+ public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
+ VectorExpression numBucketsExpression, int outputColumnNum) {
+ super(windowFrameDef, outputColumnNum);
+ if (!(numBucketsExpression instanceof ConstantVectorExpression)) {
+ throw new RuntimeException(
+ "NTILE expects a constant integer expression for the number of buckets, got: "
+ + (numBucketsExpression == null ? "null"
+ : numBucketsExpression.getClass().getName()));
+ }
+ long value = ((ConstantVectorExpression) numBucketsExpression).getLongValue();
+ if (value <= 0 || value > Integer.MAX_VALUE) {
+ throw new RuntimeException(
+ "NTILE requires a positive integer number of buckets, got: " + value);
+ }
+ this.numBuckets = (int) value;
+ resetEvaluator();
+ }
+
+ @Override
+ public boolean isGroupAggregatedStreamingEvaluator() {
+ return true;
+ }
+
+ @Override
+ public void addStreamingGroupResults(List<Integer> groupRowCounts) throws HiveException {
+ if (partitionSize < 0) {
+ throw new HiveException("Partition size must be set before computing ntile");
+ }
+ // Peer groups are irrelevant: buckets depend only on row position and partition size.
+ bucketSize = partitionSize / numBuckets;
+ largerBucketCount = partitionSize % numBuckets;
+ rowsLeftInPartition = partitionSize;
+ currentBucket = 1;
+ rowsLeftInBucket = rowsInBucket(currentBucket);
+ }
+
+ @Override
+ public void evaluateGroupBatch(VectorizedRowBatch batch) throws HiveException {
+ final int size = batch.size;
+ if (size > rowsLeftInPartition) {
+ throw new HiveException("ntile received " + size + " rows but only " + rowsLeftInPartition
+ + " remain in the partition (partitionSize=" + partitionSize + ")");
+ }
+ if (size == 0) {
Review Comment:
Done. An empty batch now writes vector[0] and uses up no rows, which is harmless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4248228270
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+ private int rowsLeftInPartition;
Review Comment:
Agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ramitg254 (via GitHub)" <gi...@apache.org>.
ramitg254 commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4245464474
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/VectorPTFDesc.java:
##########
@@ -222,6 +225,11 @@ public static VectorPTFEvaluatorBase getEvaluator(SupportedFunctionType function
case CUME_DIST:
evaluator = new VectorPTFEvaluatorCumeDist(windowFrameDef, outputColumnNum);
break;
+ case NTILE:
+ // ntile(n) receives the number of buckets as a constant integer expression
Review Comment:
please drop this comment , it requires input is obvious via use of `inputVectorExpression`
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+ private int rowsLeftInPartition;
Review Comment:
this can be dropped, the case it is checking https://github.com/apache/hive/pull/6861/changes#diff-8672208dc0cc7169c3fd2ece5a2319b7a6b470b13ebc5a54e3c539f27b1fb1c1R102
can be checked via
`bucketSize<=0`
as in other cases it can't exceed rows left in partition
and can be returned early when` bucketSize` is evaluated rather than in batch evaluation
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,385 @@
+/*
Review Comment:
the evaluation can be tested via q test this isn't needed IMHO
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+ private int rowsLeftInPartition;
+
+ public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
+ VectorExpression numBucketsExpression, int outputColumnNum) {
+ super(windowFrameDef, outputColumnNum);
+ if (!(numBucketsExpression instanceof ConstantVectorExpression)) {
+ throw new RuntimeException(
+ "NTILE expects a constant integer expression for the number of buckets, got: "
+ + (numBucketsExpression == null ? "null"
+ : numBucketsExpression.getClass().getName()));
+ }
+ long value = ((ConstantVectorExpression) numBucketsExpression).getLongValue();
+ if (value <= 0 || value > Integer.MAX_VALUE) {
+ throw new RuntimeException(
+ "NTILE requires a positive integer number of buckets, got: " + value);
+ }
+ this.numBuckets = (int) value;
+ resetEvaluator();
+ }
+
+ @Override
+ public boolean isGroupAggregatedStreamingEvaluator() {
+ return true;
+ }
+
+ @Override
+ public void addStreamingGroupResults(List<Integer> groupRowCounts) throws HiveException {
+ if (partitionSize < 0) {
+ throw new HiveException("Partition size must be set before computing ntile");
+ }
+ // Peer groups are irrelevant: buckets depend only on row position and partition size.
+ bucketSize = partitionSize / numBuckets;
+ largerBucketCount = partitionSize % numBuckets;
+ rowsLeftInPartition = partitionSize;
+ currentBucket = 1;
+ rowsLeftInBucket = rowsInBucket(currentBucket);
+ }
+
+ @Override
+ public void evaluateGroupBatch(VectorizedRowBatch batch) throws HiveException {
+ final int size = batch.size;
+ if (size > rowsLeftInPartition) {
+ throw new HiveException("ntile received " + size + " rows but only " + rowsLeftInPartition
+ + " remain in the partition (partitionSize=" + partitionSize + ")");
+ }
+ if (size == 0) {
Review Comment:
this defensive check can also be dropped
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
why we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4247431160
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
Without this change, first commit failed CI on sketches_rewrite_ntile.q and sketches_rewrite_ntile_partition_by.q with below ERROR -
``` org.apache.hadoop.hive.ql.exec.tez.TezRuntimeException: Vertex failed, vertexName=Reducer 3, vertexId=vertex_1791496978238_0001_162_03, diagnostics=[Task failed, taskId=task_1791496978238_0001_162_03_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1791496978238_0001_162_03_000000_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:342)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:86)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:72)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:42)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:112)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:411)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:259)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:292)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:294)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing vector batch (tag=0) (vectorizedVertexNum 3)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:519)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:402)
... 18 more
Caused by: java.lang.RuntimeException: No complex type support: LIST
at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.copyNonSelectedColumnVector(VectorizedBatchUtil.java:802)
at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.copyNonSelectedColumnVector(VectorizedBatchUtil.java:644)
at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.copyBufferedColumns(VectorPTFGroupBatches.java:1265)
at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.bufferGroupBatch(VectorPTFGroupBatches.java:545)
at org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFOperator.process(VectorPTFOperator.java:421)
at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:920)
at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:159)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)
... 19 more
```
This fixes sketches_rewrite_ntile.q and sketches_rewrite_ntile_partition_by.q, which failed in CI on the first commit with No complex type support: LIST.
initBufferedColumns buffered every scratch column of the reducer, including the array<double> that the Select after the PTF computes for ds_kll_cdf(...)[0], and VectorizedBatchUtil.copyNonSelectedColumnVector can't copy lists. Since HIVE-29696, only evaluator input expressions are evaluated before buffering, so those are the only scratch columns a buffered batch needs. The rest belong to child operators, which recompute them after the batch is forwarded (prepareBufferedBatchForForwarding resets the overflow batch first). It isn't ntile-specific: the same query with cume_dist() instead of ntile fails the same way. It only showed up now because vectorizing ntile sends those queries through the buffered path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ramitg254 (via GitHub)" <gi...@apache.org>.
ramitg254 commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4247597434
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/vector/ptf/TestVectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,385 @@
+/*
Review Comment:
generally qtest covers all the function related cases in execution and we don't follow convention of using junit test case for each ptf function so I think we should stick to qtest only.
do we have any explicitly behaviour which is crucial to be tested via junit test and can't be tested via q test ?
if not then we should drop this entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4248224419
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/VectorPTFDesc.java:
##########
@@ -222,6 +225,11 @@ public static VectorPTFEvaluatorBase getEvaluator(SupportedFunctionType function
case CUME_DIST:
evaluator = new VectorPTFEvaluatorCumeDist(windowFrameDef, outputColumnNum);
break;
+ case NTILE:
+ // ntile(n) receives the number of buckets as a constant integer expression
Review Comment:
Done, removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270394849
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+
+ public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
+ VectorExpression numBucketsExpression, int outputColumnNum) {
+ super(windowFrameDef, outputColumnNum);
+ if (!(numBucketsExpression instanceof ConstantVectorExpression)) {
+ throw new RuntimeException(
+ "NTILE expects a constant integer expression for the number of buckets, got: "
+ + (numBucketsExpression == null ? "null"
+ : numBucketsExpression.getClass().getName()));
+ }
+ long value = ((ConstantVectorExpression) numBucketsExpression).getLongValue();
+ if (value <= 0 || value > Integer.MAX_VALUE) {
+ throw new RuntimeException(
+ "NTILE requires a positive integer number of buckets, got: " + value);
+ }
+ this.numBuckets = (int) value;
Review Comment:
Done. numBuckets is now a long. The only int conversion left is (int) Math.min(numBuckets, partitionSize), which is bounded by the partition size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4260242058
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
Done. Removed the workaround from both files; they're now identical to their versions before HIVE-25124. Both pass with PTF vectorization on, and their .q.out files don't change.
Validated using below mvn test
```
mvn test -o -Pitests -pl itests/qtest -Dtest=TestMiniLlapLocalCliDriver -Dqfile=sketches_rewrite_cume_dist.q,sketches_rewrite_cume_dist_partition_by.q -Drat.skip=true
```
Validated below files as well, Seems these are Intentional non-vectorized reference runs, so vectorized output can be compared against them
- vector_ptf_bounded_start
- count_distinct
- cume_dist
- lead_lag
- percent_rank
- windowing_over_const
- vector_windowing_row_number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270392049
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFEvaluatorNTile.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.vector.ptf;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.hadoop.hive.ql.exec.vector.ColumnVector.Type;
+import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.ConstantVectorExpression;
+import org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpression;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.plan.ptf.WindowFrameDef;
+
+/**
+ * Evaluates {@code ntile(n)} as a <b>group-aggregated streaming</b> evaluator (see
+ * {@link VectorPTFEvaluatorBase}).
+ *
+ * <p>NTILE divides the rows of a partition into {@code n} buckets and returns the 1-based bucket
+ * number of the current row. When the partition size is not divisible by {@code n}, the first
+ * {@code partitionSize % n} buckets each get one extra row, matching {@code GenericUDAFNTile}.
+ * The result depends only on row position and partition size, never on peer groups.
+ *
+ * <p>The partition is buffered so {@link #setPartitionSize(int)} is known before output is
+ * written. Bucket boundaries are then derived on the fly while the batches are replayed, so the
+ * evaluator keeps O(1) state regardless of partition size, and a batch that falls entirely within
+ * one bucket is emitted as a repeating vector.
+ *
+ * <p>The number of buckets must be a positive integer constant (e.g. {@code ntile(4)}); the
+ * Vectorizer leaves any other form to row mode.
+ */
+public class VectorPTFEvaluatorNTile extends VectorPTFEvaluatorBase {
+
+ private final int numBuckets;
+
+ // Per-partition bucket layout, set up by addStreamingGroupResults.
+ private int bucketSize;
+ private int largerBucketCount;
+
+ // Replay position within the partition.
+ private int currentBucket;
+ private int rowsLeftInBucket;
+
+ public VectorPTFEvaluatorNTile(WindowFrameDef windowFrameDef,
+ VectorExpression numBucketsExpression, int outputColumnNum) {
+ super(windowFrameDef, outputColumnNum);
+ if (!(numBucketsExpression instanceof ConstantVectorExpression)) {
Review Comment:
Done, thanks. Used the same pattern in Vectorizer.isPositiveIntConstant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4270401531
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -474,13 +476,31 @@ private void initBufferedColumns() {
bufferedTypeInfos[bufferedMapIndex] = outputTypeInfos[streamingEvaluatorNum];
}
- for (int i = 0; i < scratchColumnPositions.length; i++) {
+ for (int i = 0; i < bufferedScratchIndices.length; i++) {
+ final int scratchIndex = bufferedScratchIndices[i];
final int bufferedMapIndex = orderColumnCount + nonKeyInputColumnCount + streamingEvaluatorCount + i;
- bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[i];
- bufferedTypeInfos[bufferedMapIndex] = TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[i]);
+ bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[scratchIndex];
+ bufferedTypeInfos[bufferedMapIndex] =
+ TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[scratchIndex]);
}
}
+ /**
+ * Returns the indices (into scratchColumnPositions) of the scratch columns written by evaluator
+ * input expressions. Those are evaluated before a batch is buffered, so their results must be
+ * buffered too. The other scratch columns belong to child operators, which recompute them after
+ * the batch is forwarded; buffering them is wasted work and fails for complex types.
+ */
+ private int[] getEvaluatorInputScratchIndices() {
Review Comment:
Correct, it's not ntile-specific.
It's a pre-existing bug in buffering any vectorized PTF (cume_dist, percent_rank, sum, …). Vectorizing ntile exposed it in sketches_rewrite_ntile*.q. I've added a separate "Related fix" section to the PR description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6138226468
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [1.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6861:
URL: https://github.com/apache/hive/pull/6861#issuecomment-6127473804
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6861&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6861&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_coverage&view=list)
 [0.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6861&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6861)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ramitg254 (via GitHub)" <gi...@apache.org>.
ramitg254 commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4255999719
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -435,10 +435,11 @@ private void initBufferedColumns() {
* 1. order columns
* 2. the non-key input columns
* 3. any streaming columns that will already have their output values
- * 4. scratch columns (as they're used by input expressions of evaluators)
+ * 4. scratch columns holding the results of evaluator input expressions
*/
+ final int[] bufferedScratchIndices = getEvaluatorInputScratchIndices();
Review Comment:
change makes sense, I know I am asking too much but can you please remove workaround added in these files:
ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist.q
ql/src/test/queries/clientpositive/sketches_rewrite_cume_dist_partition_by.q
in this change https://github.com/apache/hive/pull/6703/changes#diff-e03a7feea22c2de54e39d2a81840303b5fd18d4bdfdf15be9af2eab68984f5cb
it should also pass now without that workaround
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30018: PTF: Vectorize ntile function [hive]
Posted by "ggangadharan (via GitHub)" <gi...@apache.org>.
ggangadharan commented on code in PR #6861:
URL: https://github.com/apache/hive/pull/6861#discussion_r4260194734
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/ptf/VectorPTFOperator.java:
##########
@@ -474,13 +475,31 @@ private void initBufferedColumns() {
bufferedTypeInfos[bufferedMapIndex] = outputTypeInfos[streamingEvaluatorNum];
}
- for (int i = 0; i < scratchColumnPositions.length; i++) {
+ for (int i = 0; i < bufferedScratchIndices.length; i++) {
+ final int scratchIndex = bufferedScratchIndices[i];
final int bufferedMapIndex = orderColumnCount + nonKeyInputColumnCount + streamingEvaluatorCount + i;
- bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[i];
- bufferedTypeInfos[bufferedMapIndex] = TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[i]);
+ bufferedColumnMap[bufferedMapIndex] = scratchColumnPositions[scratchIndex];
+ bufferedTypeInfos[bufferedMapIndex] =
+ TypeInfoUtils.getTypeInfoFromTypeString(vOutContext.getScratchColumnTypeNames()[scratchIndex]);
}
}
+ /**
+ * Returns the indices (into scratchColumnPositions) of the scratch columns written by evaluator
+ * input expressions. Those are evaluated before a batch is buffered, so their results must be
+ * buffered too. The other scratch columns belong to child operators, which recompute them after
+ * the batch is forwarded; buffering them is wasted work and fails for complex types.
+ */
+ private int[] getEvaluatorInputScratchIndices() {
+ final int[] inputExprColumns = Arrays.stream(evaluators)
+ .filter(evaluator -> evaluator.inputVecExpr != null)
+ .mapToInt(evaluator -> evaluator.inputVecExpr.getOutputColumnNum())
+ .toArray();
+ return IntStream.range(0, scratchColumnPositions.length)
Review Comment:
Done. Collected the evaluator input columns into a Set (only membership is needed, no values) and filter the scratch indices with contains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org