You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by we...@apache.org on 2017/05/08 22:17:48 UTC
[31/50] [abbrv] hive git commit: HIVE-16275: Vectorization: Add
ReduceSink support for TopN (in specialized native classes) (Matt McCline,
reviewed by Gopal Vijayaraghavan)
HIVE-16275: Vectorization: Add ReduceSink support for TopN (in specialized native classes) (Matt McCline, reviewed by Gopal Vijayaraghavan)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/ec8c390e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/ec8c390e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/ec8c390e
Branch: refs/heads/hive-14535
Commit: ec8c390ee52931169d0b63bae3f8b3886170e8b8
Parents: 9d4f13a
Author: Matt McCline <mm...@hortonworks.com>
Authored: Mon May 8 00:37:48 2017 -0500
Committer: Matt McCline <mm...@hortonworks.com>
Committed: Mon May 8 00:37:48 2017 -0500
----------------------------------------------------------------------
.../VectorReduceSinkCommonOperator.java | 51 ++-
.../VectorReduceSinkObjectHashOperator.java | 1 -
.../ql/exec/vector/udf/VectorUDFAdaptor.java | 23 ++
.../hive/ql/optimizer/physical/Vectorizer.java | 9 +-
.../hadoop/hive/ql/plan/ReduceSinkDesc.java | 4 +-
.../hive/ql/plan/VectorReduceSinkDesc.java | 10 +-
.../clientpositive/vectorization_limit.q | 14 +-
.../llap/llap_vector_nohybridgrace.q.out | 8 +-
.../llap/vector_adaptor_usage_mode.q.out | 2 +-
.../llap/vector_auto_smb_mapjoin_14.q.out | 10 +-
.../llap/vector_between_columns.q.out | 4 +-
.../clientpositive/llap/vector_between_in.q.out | 32 +-
.../llap/vector_binary_join_groupby.q.out | 12 +-
.../clientpositive/llap/vector_bucket.q.out | 2 +-
.../clientpositive/llap/vector_char_2.q.out | 28 +-
.../llap/vector_char_mapjoin1.q.out | 12 +-
.../llap/vector_char_simple.q.out | 13 +-
.../clientpositive/llap/vector_coalesce.q.out | 45 +--
.../clientpositive/llap/vector_coalesce_2.q.out | 2 +-
.../clientpositive/llap/vector_count.q.out | 8 +-
.../llap/vector_count_distinct.q.out | 4 +-
.../clientpositive/llap/vector_data_types.q.out | 9 +-
.../llap/vector_decimal_aggregate.q.out | 2 +-
.../llap/vector_decimal_expressions.q.out | 9 +-
.../llap/vector_decimal_mapjoin.q.out | 2 +-
.../llap/vector_decimal_round.q.out | 8 +-
.../llap/vector_decimal_round_2.q.out | 8 +-
.../clientpositive/llap/vector_distinct_2.q.out | 2 +-
.../llap/vector_empty_where.q.out | 16 +-
.../clientpositive/llap/vector_groupby4.q.out | 4 +-
.../clientpositive/llap/vector_groupby6.q.out | 4 +-
.../clientpositive/llap/vector_groupby_3.q.out | 2 +-
.../llap/vector_groupby_grouping_id3.q.out | 4 +-
.../llap/vector_groupby_mapjoin.q.out | 10 +-
.../llap/vector_groupby_reduce.q.out | 26 +-
.../llap/vector_grouping_sets.q.out | 4 +-
.../clientpositive/llap/vector_if_expr.q.out | 2 +-
.../llap/vector_include_no_sel.q.out | 4 +-
.../clientpositive/llap/vector_inner_join.q.out | 18 +-
.../clientpositive/llap/vector_interval_1.q.out | 16 +-
.../clientpositive/llap/vector_interval_2.q.out | 20 +-
.../llap/vector_interval_arithmetic.q.out | 24 +-
.../llap/vector_interval_mapjoin.q.out | 2 +-
.../clientpositive/llap/vector_join30.q.out | 84 ++---
.../llap/vector_join_part_col_char.q.out | 18 +-
.../llap/vector_left_outer_join2.q.out | 8 +-
.../llap/vector_leftsemi_mapjoin.q.out | 376 +++++++++----------
.../llap/vector_mapjoin_reduce.q.out | 12 +-
.../llap/vector_non_constant_in_expr.q.out | 4 +-
.../llap/vector_non_string_partition.q.out | 18 +-
.../llap/vector_nullsafe_join.q.out | 36 +-
.../llap/vector_number_compare_projection.q.out | 4 +-
.../clientpositive/llap/vector_order_null.q.out | 22 +-
.../clientpositive/llap/vector_orderby_5.q.out | 4 +-
.../llap/vector_outer_join0.q.out | 4 +-
.../llap/vector_outer_join1.q.out | 10 +-
.../llap/vector_outer_join2.q.out | 6 +-
.../llap/vector_partition_diff_num_cols.q.out | 10 +-
.../llap/vector_partitioned_date_time.q.out | 60 ++-
.../llap/vector_ptf_part_simple.q.out | 30 +-
.../clientpositive/llap/vector_reduce1.q.out | 2 +-
.../clientpositive/llap/vector_reduce2.q.out | 2 +-
.../clientpositive/llap/vector_reduce3.q.out | 2 +-
.../llap/vector_reduce_groupby_decimal.q.out | 14 +-
.../llap/vector_string_concat.q.out | 14 +-
.../llap/vector_string_decimal.q.out | 24 +-
.../llap/vector_tablesample_rows.q.out | 2 +-
.../llap/vector_varchar_simple.q.out | 13 +-
.../llap/vector_when_case_null.q.out | 2 +-
.../clientpositive/llap/vectorization_0.q.out | 24 +-
.../clientpositive/llap/vectorization_7.q.out | 18 +-
.../clientpositive/llap/vectorization_8.q.out | 18 +-
.../llap/vectorization_div0.q.out | 18 +-
.../llap/vectorization_limit.q.out | 331 +++++++++-------
.../llap/vectorization_offset_limit.q.out | 9 +-
.../llap/vectorization_part_project.q.out | 2 +-
.../llap/vectorization_short_regress.q.out | 52 ++-
.../llap/vectorized_bucketmapjoin1.q.out | 10 +-
.../clientpositive/llap/vectorized_case.q.out | 4 +-
.../llap/vectorized_date_funcs.q.out | 4 +-
.../vectorized_dynamic_semijoin_reduction.q.out | 58 +--
.../llap/vectorized_mapjoin.q.out | 2 +-
.../llap/vectorized_mapjoin2.q.out | 4 +-
.../clientpositive/llap/vectorized_ptf.q.out | 50 +--
.../llap/vectorized_shufflejoin.q.out | 4 +-
.../llap/vectorized_timestamp_funcs.q.out | 10 +-
.../spark/vector_between_in.q.out | 32 +-
.../spark/vector_count_distinct.q.out | 4 +-
.../spark/vector_data_types.q.out | 9 +-
.../spark/vector_decimal_aggregate.q.out | 2 +-
.../spark/vector_distinct_2.q.out | 2 +-
.../clientpositive/spark/vector_groupby_3.q.out | 2 +-
.../spark/vector_mapjoin_reduce.q.out | 4 +-
.../clientpositive/spark/vector_orderby_5.q.out | 4 +-
.../spark/vector_outer_join1.q.out | 2 +-
.../spark/vector_outer_join2.q.out | 2 +-
.../spark/vector_string_concat.q.out | 14 +-
.../clientpositive/spark/vectorization_0.q.out | 24 +-
.../clientpositive/spark/vectorization_7.q.out | 18 +-
.../clientpositive/spark/vectorization_8.q.out | 18 +-
.../spark/vectorization_div0.q.out | 18 +-
.../spark/vectorization_part_project.q.out | 2 +-
.../spark/vectorization_short_regress.q.out | 52 ++-
.../clientpositive/spark/vectorized_case.q.out | 4 +-
.../clientpositive/spark/vectorized_ptf.q.out | 50 +--
.../spark/vectorized_shufflejoin.q.out | 4 +-
.../spark/vectorized_timestamp_funcs.q.out | 10 +-
.../tez/vector_non_string_partition.q.out | 18 +-
.../clientpositive/tez/vectorization_div0.q.out | 18 +-
.../tez/vectorization_limit.q.out | 281 ++++++++------
.../vector_binary_join_groupby.q.out | 8 +-
.../results/clientpositive/vector_bucket.q.out | 2 +-
.../clientpositive/vector_cast_constant.q.out | 4 +-
.../results/clientpositive/vector_char_2.q.out | 16 +-
.../clientpositive/vector_char_mapjoin1.q.out | 6 +-
.../clientpositive/vector_char_simple.q.out | 4 +-
.../clientpositive/vector_coalesce.q.out | 20 +-
.../clientpositive/vector_coalesce_2.q.out | 2 +-
.../results/clientpositive/vector_count.q.out | 8 +-
.../clientpositive/vector_data_types.q.out | 4 +-
.../vector_decimal_aggregate.q.out | 2 +-
.../vector_decimal_expressions.q.out | 4 +-
.../clientpositive/vector_decimal_round.q.out | 8 +-
.../clientpositive/vector_decimal_round_2.q.out | 8 +-
.../clientpositive/vector_distinct_2.q.out | 2 +-
.../clientpositive/vector_empty_where.q.out | 8 +-
.../clientpositive/vector_groupby4.q.out | 4 +-
.../clientpositive/vector_groupby6.q.out | 4 +-
.../clientpositive/vector_groupby_3.q.out | 2 +-
.../clientpositive/vector_groupby_mapjoin.q.out | 6 +-
.../clientpositive/vector_groupby_reduce.q.out | 20 +-
.../clientpositive/vector_grouping_sets.q.out | 4 +-
.../results/clientpositive/vector_if_expr.q.out | 2 +-
.../clientpositive/vector_include_no_sel.q.out | 2 +-
.../clientpositive/vector_interval_1.q.out | 16 +-
.../vector_interval_arithmetic.q.out | 16 +-
.../clientpositive/vector_mapjoin_reduce.q.out | 4 +-
.../vector_non_string_partition.q.out | 8 +-
.../clientpositive/vector_order_null.q.out | 22 +-
.../clientpositive/vector_orderby_5.q.out | 4 +-
.../clientpositive/vector_outer_join1.q.out | 2 +-
.../clientpositive/vector_outer_join2.q.out | 2 +-
.../clientpositive/vector_outer_join3.q.out | 6 +-
.../clientpositive/vector_outer_join4.q.out | 2 +-
.../results/clientpositive/vector_reduce1.q.out | 2 +-
.../results/clientpositive/vector_reduce2.q.out | 2 +-
.../results/clientpositive/vector_reduce3.q.out | 2 +-
.../vector_reduce_groupby_decimal.q.out | 8 +-
.../clientpositive/vector_string_concat.q.out | 8 +-
.../vector_tablesample_rows.q.out | 2 +-
.../clientpositive/vector_varchar_simple.q.out | 4 +-
.../clientpositive/vector_when_case_null.q.out | 2 +-
.../clientpositive/vectorization_13.q.out | 8 +-
.../clientpositive/vectorization_7.q.out | 8 +-
.../clientpositive/vectorization_8.q.out | 8 +-
.../clientpositive/vectorization_div0.q.out | 8 +-
.../clientpositive/vectorization_limit.q.out | 247 ++++++------
.../vectorization_offset_limit.q.out | 4 +-
.../clientpositive/vectorized_case.q.out | 4 +-
.../clientpositive/vectorized_date_funcs.q.out | 4 +-
.../clientpositive/vectorized_mapjoin2.q.out | 2 +-
.../vectorized_parquet_types.q.out | 2 +-
.../clientpositive/vectorized_shufflejoin.q.out | 2 +-
.../clientpositive/vectorized_timestamp.q.out | 2 +-
.../vectorized_timestamp_funcs.q.out | 10 +-
165 files changed, 1609 insertions(+), 1445 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
index fc5aea5..a4dbe0b 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.hive.ql.exec.vector.reducesink;
import java.io.IOException;
+import java.io.Serializable;
import java.util.Arrays;
import java.util.Properties;
@@ -29,6 +30,7 @@ import org.apache.hadoop.hive.ql.CompilationOpContext;
import org.apache.hadoop.hive.ql.exec.Operator;
import org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.Counter;
import org.apache.hadoop.hive.ql.exec.TerminalOperator;
+import org.apache.hadoop.hive.ql.exec.TopNHash;
import org.apache.hadoop.hive.ql.exec.Utilities;
import org.apache.hadoop.hive.ql.exec.vector.VectorSerializeRow;
import org.apache.hadoop.hive.ql.exec.vector.VectorizationContext;
@@ -57,11 +59,13 @@ import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hive.common.util.HashCodeUtil;
+import com.google.common.base.Preconditions;
+
/**
* This class is common operator class for native vectorized reduce sink.
*/
public abstract class VectorReduceSinkCommonOperator extends TerminalOperator<ReduceSinkDesc>
- implements VectorizationContextRegion {
+ implements Serializable, TopNHash.BinaryCollector, VectorizationContextRegion {
private static final long serialVersionUID = 1L;
private static final String CLASS_NAME = VectorReduceSinkCommonOperator.class.getName();
@@ -122,6 +126,9 @@ public abstract class VectorReduceSinkCommonOperator extends TerminalOperator<Re
protected transient HiveKey keyWritable;
protected transient BytesWritable valueBytesWritable;
+ // Picks topN K:V pairs from input.
+ protected transient TopNHash reducerHash;
+
// Where to write our key and value pairs.
private transient OutputCollector out;
@@ -329,10 +336,46 @@ public abstract class VectorReduceSinkCommonOperator extends TerminalOperator<Re
valueBytesWritable = new BytesWritable();
+ int limit = conf.getTopN();
+ float memUsage = conf.getTopNMemoryUsage();
+
+ if (limit >= 0 && memUsage > 0) {
+ reducerHash = new TopNHash();
+ reducerHash.initialize(limit, memUsage, conf.isMapGroupBy(), this, conf, hconf);
+ }
+
batchCounter = 0;
}
- protected void collect(BytesWritable keyWritable, Writable valueWritable) throws IOException {
+ // The collect method override for TopNHash.BinaryCollector
+ @Override
+ public void collect(byte[] key, byte[] value, int hash) throws IOException {
+ HiveKey keyWritable = new HiveKey(key, hash);
+ BytesWritable valueWritable = new BytesWritable(value);
+ doCollect(keyWritable, valueWritable);
+ }
+
+ protected void collect(HiveKey keyWritable, BytesWritable valueWritable)
+ throws HiveException, IOException {
+ if (reducerHash != null) {
+ // NOTE: partColsIsNull is only used for PTF, which isn't supported yet.
+ final int firstIndex =
+ reducerHash.tryStoreKey(keyWritable, /* partColsIsNull */ false);
+
+ if (firstIndex == TopNHash.EXCLUDE) return; // Nothing to do.
+
+ if (firstIndex == TopNHash.FORWARD) {
+ doCollect(keyWritable, valueWritable);
+ } else {
+ Preconditions.checkState(firstIndex >= 0);
+ reducerHash.storeValue(firstIndex, keyWritable.hashCode(), valueWritable, false);
+ }
+ } else {
+ doCollect(keyWritable, valueWritable);
+ }
+ }
+
+ private void doCollect(HiveKey keyWritable, BytesWritable valueWritable) throws IOException {
// Since this is a terminal operator, update counters explicitly -
// forward is not called
if (null != out) {
@@ -360,8 +403,12 @@ public abstract class VectorReduceSinkCommonOperator extends TerminalOperator<Re
@Override
protected void closeOp(boolean abort) throws HiveException {
+ if (!abort && reducerHash != null) {
+ reducerHash.flush();
+ }
super.closeOp(abort);
out = null;
+ reducerHash = null;
if (isLogInfoEnabled) {
LOG.info(toString() + ": records written - " + numRows);
}
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
index 6312c44..bd7d6cb 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java
@@ -177,7 +177,6 @@ public class VectorReduceSinkObjectHashOperator extends VectorReduceSinkCommonOp
nonPartitionRandom = new Random(12345);
} else {
partitionObjectInspectors = getObjectInspectorArray(reduceSinkPartitionTypeInfos);
- LOG.debug("*NEW* partitionObjectInspectors " + Arrays.toString(partitionObjectInspectors));
partitionVectorExtractRow = new VectorExtractRow();
partitionVectorExtractRow.init(reduceSinkPartitionTypeInfos, reduceSinkPartitionColumnMap);
partitionFieldValues = new Object[reduceSinkPartitionTypeInfos.length];
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/udf/VectorUDFAdaptor.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/udf/VectorUDFAdaptor.java b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/udf/VectorUDFAdaptor.java
index 0bb0f22..3b3624d 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/udf/VectorUDFAdaptor.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/udf/VectorUDFAdaptor.java
@@ -20,6 +20,8 @@ package org.apache.hadoop.hive.ql.exec.vector.udf;
import java.sql.Date;
import java.sql.Timestamp;
+import org.apache.hadoop.hive.common.type.HiveIntervalYearMonth;
+import org.apache.hadoop.hive.common.type.HiveIntervalDayTime;
import org.apache.hadoop.hive.common.type.HiveDecimal;
import org.apache.hadoop.hive.ql.exec.MapredContext;
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
@@ -39,6 +41,8 @@ import org.apache.hadoop.hive.serde2.objectinspector.primitive.*;
import org.apache.hadoop.hive.serde2.typeinfo.CharTypeInfo;
import org.apache.hadoop.hive.serde2.typeinfo.VarcharTypeInfo;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableBinaryObjectInspector;
+import org.apache.hadoop.hive.serde2.io.HiveIntervalDayTimeWritable;
+import org.apache.hadoop.hive.serde2.io.HiveIntervalYearMonthWritable;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
@@ -336,6 +340,25 @@ public class VectorUDFAdaptor extends VectorExpression {
BytesWritable bw = (BytesWritable) value;
BytesColumnVector bv = (BytesColumnVector) colVec;
bv.setVal(i, bw.getBytes(), 0, bw.getLength());
+ } else if (outputOI instanceof WritableHiveIntervalYearMonthObjectInspector) {
+ LongColumnVector lv = (LongColumnVector) colVec;
+ HiveIntervalYearMonth iym;
+ if (value instanceof HiveIntervalYearMonth) {
+ iym = (HiveIntervalYearMonth) value;
+ } else {
+ iym = ((WritableHiveIntervalYearMonthObjectInspector) outputOI).getPrimitiveJavaObject(value);
+ }
+ long l = iym.getTotalMonths();
+ lv.vector[i] = l;
+ } else if (outputOI instanceof WritableHiveIntervalDayTimeObjectInspector) {
+ IntervalDayTimeColumnVector idtv = (IntervalDayTimeColumnVector) colVec;
+ HiveIntervalDayTime idt;
+ if (value instanceof HiveIntervalDayTime) {
+ idt = (HiveIntervalDayTime) value;
+ } else {
+ idt = ((WritableHiveIntervalDayTimeObjectInspector) outputOI).getPrimitiveJavaObject(value);
+ }
+ idtv.set(i, idt);
} else {
throw new RuntimeException("Unhandled object type " + outputOI.getTypeName() +
" inspector class " + outputOI.getClass().getName() +
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
index f0df2e9..2025c24 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java
@@ -3025,7 +3025,10 @@ public class Vectorizer implements PhysicalPlanResolver {
String engine = HiveConf.getVar(hiveConf, HiveConf.ConfVars.HIVE_EXECUTION_ENGINE);
- boolean hasTopN = (desc.getTopN() >= 0);
+ int limit = desc.getTopN();
+ float memUsage = desc.getTopNMemoryUsage();
+
+ boolean hasPTFTopN = (limit >= 0 && memUsage > 0 && desc.isPTFReduceSink());
boolean hasDistinctColumns = (desc.getDistinctColumnIndices().size() > 0);
@@ -3202,7 +3205,7 @@ public class Vectorizer implements PhysicalPlanResolver {
// Remember the condition variables for EXPLAIN regardless.
vectorDesc.setIsVectorizationReduceSinkNativeEnabled(isVectorizationReduceSinkNativeEnabled);
vectorDesc.setEngine(engine);
- vectorDesc.setHasTopN(hasTopN);
+ vectorDesc.setHasPTFTopN(hasPTFTopN);
vectorDesc.setHasDistinctColumns(hasDistinctColumns);
vectorDesc.setIsKeyBinarySortable(isKeyBinarySortable);
vectorDesc.setIsValueLazyBinary(isValueLazyBinary);
@@ -3215,7 +3218,7 @@ public class Vectorizer implements PhysicalPlanResolver {
if (!isVectorizationReduceSinkNativeEnabled ||
!isTezOrSpark ||
(useUniformHash && (hasEmptyBuckets || hasNoPartitions)) ||
- hasTopN ||
+ hasPTFTopN ||
hasDistinctColumns ||
!isKeyBinarySortable ||
!isValueLazyBinary ||
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java b/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
index 79d19b5..c059db6 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java
@@ -544,8 +544,8 @@ public class ReduceSinkDesc extends AbstractOperatorDesc {
engineInSupported,
engineInSupportedCondName),
new VectorizationCondition(
- !vectorReduceSinkDesc.getHasTopN(),
- "No TopN"),
+ !vectorReduceSinkDesc.getHasPTFTopN(),
+ "No PTF TopN"),
new VectorizationCondition(
!vectorReduceSinkDesc.getHasDistinctColumns(),
"No DISTINCT columns"),
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java b/ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java
index d6230af..91d5be7 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java
@@ -64,7 +64,7 @@ public class VectorReduceSinkDesc extends AbstractVectorDesc {
private boolean isVectorizationReduceSinkNativeEnabled;
private String engine;
- private boolean hasTopN;
+ private boolean hasPTFTopN;
private boolean hasDistinctColumns;
private boolean isKeyBinarySortable;
private boolean isValueLazyBinary;
@@ -85,11 +85,11 @@ public class VectorReduceSinkDesc extends AbstractVectorDesc {
public String getEngine() {
return engine;
}
- public void setHasTopN(boolean hasTopN) {
- this.hasTopN = hasTopN;
+ public void setHasPTFTopN(boolean hasPTFTopN) {
+ this.hasPTFTopN = hasPTFTopN;
}
- public boolean getHasTopN() {
- return hasTopN;
+ public boolean getHasPTFTopN() {
+ return hasPTFTopN;
}
public void setHasDistinctColumns(boolean hasDistinctColumns) {
this.hasDistinctColumns = hasDistinctColumns;
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/queries/clientpositive/vectorization_limit.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/vectorization_limit.q b/ql/src/test/queries/clientpositive/vectorization_limit.q
index a4c54f2..8044484 100644
--- a/ql/src/test/queries/clientpositive/vectorization_limit.q
+++ b/ql/src/test/queries/clientpositive/vectorization_limit.q
@@ -3,6 +3,8 @@ set hive.explain.user=false;
SET hive.vectorized.execution.enabled=true;
set hive.fetch.task.conversion=none;
+-- SORT_QUERY_RESULTS
+
explain vectorization SELECT cbigint, cdouble FROM alltypesorc WHERE cbigint < cdouble and cint > 0 limit 7;
SELECT cbigint, cdouble FROM alltypesorc WHERE cbigint < cdouble and cint > 0 limit 7;
@@ -11,31 +13,31 @@ set hive.limit.pushdown.memory.usage=0.3f;
-- HIVE-3562 Some limit can be pushed down to map stage - c/p parts from limit_pushdown
-explain vectorization expression
+explain vectorization detail
select ctinyint,cdouble,csmallint from alltypesorc where ctinyint is not null order by ctinyint,cdouble limit 20;
select ctinyint,cdouble,csmallint from alltypesorc where ctinyint is not null order by ctinyint,cdouble limit 20;
-- deduped RS
-explain vectorization expression
+explain vectorization detail
select ctinyint,avg(cdouble + 1) from alltypesorc group by ctinyint order by ctinyint limit 20;
select ctinyint,avg(cdouble + 1) from alltypesorc group by ctinyint order by ctinyint limit 20;
-- distincts
-explain vectorization expression
+explain vectorization detail
select distinct(ctinyint) from alltypesorc limit 20;
select distinct(ctinyint) from alltypesorc limit 20;
-explain vectorization expression
+explain vectorization detail
select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint order by ctinyint limit 20;
select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint order by ctinyint limit 20;
-- limit zero
-explain vectorization expression
+explain vectorization detail
select ctinyint,cdouble from alltypesorc order by ctinyint limit 0;
select ctinyint,cdouble from alltypesorc order by ctinyint limit 0;
-- 2MR (applied to last RS)
-explain vectorization expression
+explain vectorization detail
select cdouble, sum(ctinyint) as sum from alltypesorc where ctinyint is not null group by cdouble order by sum, cdouble limit 20;
select cdouble, sum(ctinyint) as sum from alltypesorc where ctinyint is not null group by cdouble order by sum, cdouble limit 20;
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/llap_vector_nohybridgrace.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/llap_vector_nohybridgrace.q.out b/ql/src/test/results/clientpositive/llap/llap_vector_nohybridgrace.q.out
index 526662d..57ddc96 100644
--- a/ql/src/test/results/clientpositive/llap/llap_vector_nohybridgrace.q.out
+++ b/ql/src/test/results/clientpositive/llap/llap_vector_nohybridgrace.q.out
@@ -83,7 +83,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
@@ -126,7 +126,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12288 Data size: 36696 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -261,7 +261,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
@@ -304,7 +304,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12288 Data size: 36696 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized, llap
LLAP IO: all inputs
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_adaptor_usage_mode.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_adaptor_usage_mode.q.out b/ql/src/test/results/clientpositive/llap/vector_adaptor_usage_mode.q.out
index 5b17144..a2ce365 100644
--- a/ql/src/test/results/clientpositive/llap/vector_adaptor_usage_mode.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_adaptor_usage_mode.q.out
@@ -1101,7 +1101,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 5 Data size: 452 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out b/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
index a0ac248..3b44bc3 100644
--- a/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
@@ -287,7 +287,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Reducer 3
@@ -531,7 +531,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 3
@@ -586,7 +586,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
@@ -1292,7 +1292,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -1335,7 +1335,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_between_columns.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_between_columns.q.out b/ql/src/test/results/clientpositive/llap/vector_between_columns.q.out
index cb9674c..d6f6ec8 100644
--- a/ql/src/test/results/clientpositive/llap/vector_between_columns.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_between_columns.q.out
@@ -164,7 +164,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 5 Data size: 36 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: smallint)
Execution mode: vectorized, llap
@@ -331,7 +331,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 5 Data size: 36 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: smallint)
Execution mode: vectorized, llap
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_between_in.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_between_in.q.out b/ql/src/test/results/clientpositive/llap/vector_between_in.q.out
index 9b58d47..ae1b2d2 100644
--- a/ql/src/test/results/clientpositive/llap/vector_between_in.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_between_in.q.out
@@ -61,7 +61,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 6144 Data size: 1233808 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -165,7 +165,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
@@ -265,7 +265,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 6144 Data size: 1233808 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -369,7 +369,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
@@ -469,7 +469,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1365 Data size: 274112 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -565,7 +565,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 10923 Data size: 2193503 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -661,7 +661,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1365 Data size: 274112 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
LLAP IO: all inputs
@@ -765,7 +765,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
@@ -1119,7 +1119,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12288 Data size: 2467616 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
@@ -1161,7 +1161,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 6144 Data size: 1233808 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 3
@@ -1257,7 +1257,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12288 Data size: 2467616 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
@@ -1299,7 +1299,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 6144 Data size: 1233808 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 3
@@ -1395,7 +1395,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12288 Data size: 2467616 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
@@ -1437,7 +1437,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 6144 Data size: 1233808 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 3
@@ -1533,7 +1533,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12288 Data size: 2467616 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
@@ -1575,7 +1575,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 6144 Data size: 1233808 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 3
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out b/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
index 339ec2c..a35659a 100644
--- a/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
@@ -185,7 +185,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
@@ -228,7 +228,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 100 Data size: 29638 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col5 (type: double), _col6 (type: boolean), _col7 (type: string), _col8 (type: timestamp), _col9 (type: decimal(4,2))
Execution mode: vectorized, llap
@@ -268,7 +268,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reducer 3
Execution mode: vectorized, llap
@@ -383,7 +383,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 100 Data size: 29638 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
@@ -433,7 +433,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 50 Data size: 14819 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Reducer 3
@@ -630,7 +630,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkLongOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 100 Data size: 29638 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: binary)
Execution mode: vectorized, llap
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_bucket.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_bucket.q.out b/ql/src/test/results/clientpositive/llap/vector_bucket.q.out
index 7b57223..6dd0cfb 100644
--- a/ql/src/test/results/clientpositive/llap/vector_bucket.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_bucket.q.out
@@ -52,7 +52,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 1 Data size: 26 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string), _col1 (type: string)
Execution mode: vectorized, llap
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_char_2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_char_2.q.out b/ql/src/test/results/clientpositive/llap/vector_char_2.q.out
index c330097..65fafb0 100644
--- a/ql/src/test/results/clientpositive/llap/vector_char_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_char_2.q.out
@@ -111,10 +111,9 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: char(20))
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkStringOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 500 Data size: 99000 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col1 (type: bigint), _col2 (type: bigint)
@@ -155,10 +154,9 @@ STAGE PLANS:
key expressions: _col0 (type: char(20))
sort order: +
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 250 Data size: 49500 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col1 (type: bigint), _col2 (type: bigint)
@@ -309,10 +307,9 @@ STAGE PLANS:
sort order: -
Map-reduce partition columns: _col0 (type: char(20))
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkStringOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 500 Data size: 99000 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col1 (type: bigint), _col2 (type: bigint)
@@ -353,10 +350,9 @@ STAGE PLANS:
key expressions: _col0 (type: char(20))
sort order: -
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 250 Data size: 49500 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col1 (type: bigint), _col2 (type: bigint)
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_char_mapjoin1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_char_mapjoin1.q.out b/ql/src/test/results/clientpositive/llap/vector_char_mapjoin1.q.out
index e0df3c0..ab67adc 100644
--- a/ql/src/test/results/clientpositive/llap/vector_char_mapjoin1.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_char_mapjoin1.q.out
@@ -188,7 +188,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 3 Data size: 323 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: char(10)), _col2 (type: int), _col3 (type: char(10))
Execution mode: vectorized, llap
@@ -231,7 +231,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 3 Data size: 294 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Execution mode: vectorized, llap
@@ -343,7 +343,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 3 Data size: 294 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Execution mode: vectorized, llap
@@ -399,7 +399,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 3 Data size: 323 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: char(10)), _col2 (type: int), _col3 (type: char(20))
Execution mode: vectorized, llap
@@ -527,7 +527,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkObjectHashOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 3 Data size: 323 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: char(10)), _col2 (type: int), _col3 (type: string)
Execution mode: vectorized, llap
@@ -570,7 +570,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 3 Data size: 273 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Execution mode: vectorized, llap
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_char_simple.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_char_simple.q.out b/ql/src/test/results/clientpositive/llap/vector_char_simple.q.out
index d3ab3f2..5e1dea8 100644
--- a/ql/src/test/results/clientpositive/llap/vector_char_simple.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_char_simple.q.out
@@ -77,7 +77,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
@@ -163,7 +163,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
@@ -251,10 +251,9 @@ STAGE PLANS:
className: VectorLimitOperator
native: true
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -262,7 +261,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_coalesce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_coalesce.q.out b/ql/src/test/results/clientpositive/llap/vector_coalesce.q.out
index 578f849..358d8ae 100644
--- a/ql/src/test/results/clientpositive/llap/vector_coalesce.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_coalesce.q.out
@@ -39,10 +39,9 @@ STAGE PLANS:
projectedOutputColumns: [6, 2, 4, 1, 16]
selectExpressions: VectorCoalesce(columns [12, 6, 13, 14, 15])(children: ConstantVectorExpression(val null) -> 12:string, col 6, CastLongToString(col 2) -> 13:String, VectorUDFAdaptor(null(cfloat)) -> 14:string, CastLongToString(col 1) -> 15:String) -> 16:string
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -50,7 +49,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: true
vectorized: true
Reducer 2
@@ -145,10 +144,9 @@ STAGE PLANS:
projectedOutputColumns: [5, 2, 15]
selectExpressions: VectorCoalesce(columns [12, 14, 13])(children: ConstantVectorExpression(val null) -> 12:double, DoubleColAddDoubleColumn(col 5, col 13)(children: FuncLog2LongToDouble(col 2) -> 13:double) -> 14:double, ConstantVectorExpression(val 0.0) -> 13:double) -> 15:double
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -156,7 +154,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
@@ -250,10 +248,9 @@ STAGE PLANS:
native: true
projectedOutputColumns: []
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -261,7 +258,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
@@ -356,10 +353,9 @@ STAGE PLANS:
projectedOutputColumns: [8, 9, 12]
selectExpressions: VectorCoalesce(columns [8, 9])(children: col 8, col 9) -> 12:timestamp
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -367,7 +363,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
@@ -460,10 +456,9 @@ STAGE PLANS:
native: true
projectedOutputColumns: []
Reduce Sink Vectorization:
- className: VectorReduceSinkOperator
- native: false
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- nativeConditionsNotMet: No TopN IS false
+ className: VectorReduceSinkObjectHashOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -471,7 +466,7 @@ STAGE PLANS:
enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
groupByVectorOutput: true
inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
- allNative: false
+ allNative: true
usesVectorUDFAdaptor: false
vectorized: true
Reducer 2
http://git-wip-us.apache.org/repos/asf/hive/blob/ec8c390e/ql/src/test/results/clientpositive/llap/vector_coalesce_2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_coalesce_2.q.out b/ql/src/test/results/clientpositive/llap/vector_coalesce_2.q.out
index 2a3eff5..d57d39f 100644
--- a/ql/src/test/results/clientpositive/llap/vector_coalesce_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_coalesce_2.q.out
@@ -241,7 +241,7 @@ STAGE PLANS:
Reduce Sink Vectorization:
className: VectorReduceSinkStringOperator
native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 4 Data size: 510 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap