You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/05/15 17:08:15 UTC
[incubator-doris] branch dev-1.0.1 updated (973153fe9b -> 162b8492e1)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
from 973153fe9b [fixbug][metadata] catalog could not load from image (#9364)
new b93bf78e4e [fix](rewrite) The where condition cannot be pushed down because there is no derivation (#8980)
new af5ff046d1 [Enhancement] [Vectorized] Refactor and optimize BinaryOperation (#9087)
new 355216f0ed [fix][feature](Function) fix return type && support hll_union_agg/group_concat agg to window function (#9119)
new 77932f066c [enhancement](load) optimize load string data and dict page write (#9123)
new 162b8492e1 [Bug] (fix) DeltaWriter::mem_consumption() coredump (#9245)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/olap/delta_writer.cpp | 9 +
be/src/olap/delta_writer.h | 7 +
be/src/olap/memtable.cpp | 83 +-
be/src/olap/memtable.h | 10 +-
be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 34 +-
be/src/olap/rowset/segment_v2/binary_plain_page.h | 18 +-
be/src/olap/rowset/segment_v2/bitshuffle_page.h | 37 +-
be/src/runtime/tablets_channel.cpp | 8 +-
.../vec/aggregate_functions/aggregate_function.h | 2 +-
.../aggregate_function_hll_union_agg.h | 4 +
.../aggregate_function_stddev.h | 11 +-
be/src/vec/data_types/data_type_decimal.h | 6 +-
be/src/vec/data_types/number_traits.h | 89 +--
be/src/vec/functions/divide.cpp | 34 +-
be/src/vec/functions/function.h | 9 -
be/src/vec/functions/function_binary_arithmetic.h | 881 ++++++++++++---------
.../function_binary_arithmetic_to_null_type.h | 247 ------
be/src/vec/functions/function_bit.cpp | 8 +-
be/src/vec/functions/function_cast.h | 1 -
be/src/vec/functions/int_div.cpp | 118 +--
be/src/vec/functions/int_div.h | 49 +-
be/src/vec/functions/math.cpp | 39 +-
be/src/vec/functions/minus.cpp | 2 +-
be/src/vec/functions/modulo.cpp | 180 ++---
be/src/vec/functions/multiply.cpp | 2 +-
be/src/vec/functions/plus.cpp | 2 +-
.../java/org/apache/doris/catalog/FunctionSet.java | 8 +-
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../org/apache/doris/rewrite/InferFiltersRule.java | 12 +-
.../apache/doris/rewrite/InferFiltersRuleTest.java | 21 +
...t_constant.out => test_select_variance_agg.out} | 7 +-
.../correctness/test_select_variance_agg.groovy | 41 +
32 files changed, 901 insertions(+), 1080 deletions(-)
delete mode 100644 be/src/vec/functions/function_binary_arithmetic_to_null_type.h
copy regression-test/data/correctness/{test_select_constant.out => test_select_variance_agg.out} (58%)
create mode 100644 regression-test/suites/correctness/test_select_variance_agg.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 01/05: [fix](rewrite) The where condition cannot be pushed down because there is no derivation (#8980)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit b93bf78e4edf156cac499b22d301e166c3cc3253
Author: zhengshiJ <32...@users.noreply.github.com>
AuthorDate: Sat May 7 20:41:11 2022 +0800
[fix](rewrite) The where condition cannot be pushed down because there is no derivation (#8980)
Fix a bug.
The where condition cannot be pushed down because there is no derivation
eg:
select * from tb1 left join tb2 on tb1.id = tb2.id where tb2.id = 1;
The correct case is that the condition of "=1" needs to be deduced to tb1.id,
but the current implementation does not do the deduction
---
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../org/apache/doris/rewrite/InferFiltersRule.java | 12 ++++--------
.../apache/doris/rewrite/InferFiltersRuleTest.java | 21 +++++++++++++++++++++
3 files changed, 26 insertions(+), 9 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 20fccab3fa..8bc47eac47 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -422,7 +422,7 @@ public class SessionVariable implements Serializable, Writable {
private boolean disableJoinReorder = false;
@VariableMgr.VarAttr(name = ENABLE_INFER_PREDICATE)
- private boolean enableInferPredicate = false;
+ private boolean enableInferPredicate = true;
@VariableMgr.VarAttr(name = SQL_QUOTE_SHOW_CREATE)
public boolean sqlQuoteShowCreate = true;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java b/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java
index 6cfc4cac45..3ccd390144 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/InferFiltersRule.java
@@ -70,16 +70,14 @@ public class InferFiltersRule implements ExprRewriteRule {
}
// slotEqSlotExpr: Record existing and infer equivalent connections
- List<Expr> slotEqSlotExpr =
- (clauseType == ExprRewriter.ClauseType.ON_CLAUSE) ? analyzer.getOnSlotEqSlotExpr() : new ArrayList<>();
+ List<Expr> slotEqSlotExpr = analyzer.getOnSlotEqSlotExpr();
// slotEqSlotDeDuplication: De-Duplication for slotEqSlotExpr
Set<Pair<Expr, Expr>> slotEqSlotDeDuplication =
(clauseType == ExprRewriter.ClauseType.ON_CLAUSE) ? analyzer.getOnSlotEqSlotDeDuplication() : Sets.newHashSet();
// slotToLiteralExpr: Record existing and infer expr which slot and literal are equal
- List<Expr> slotToLiteralExpr =
- (clauseType == ExprRewriter.ClauseType.ON_CLAUSE) ? analyzer.getOnSlotToLiteralExpr() : new ArrayList<>();
+ List<Expr> slotToLiteralExpr = analyzer.getOnSlotToLiteralExpr();
// slotToLiteralDeDuplication: De-Duplication for slotToLiteralExpr
Set<Pair<Expr, Expr>> slotToLiteralDeDuplication =
@@ -91,16 +89,14 @@ public class InferFiltersRule implements ExprRewriteRule {
List<Pair<Expr, Boolean>> newExprWithState = new ArrayList<>();
// isNullExpr: Record existing and infer not null predicate
- List<Expr> isNullExpr =
- (clauseType == ExprRewriter.ClauseType.ON_CLAUSE) ? analyzer.getOnIsNullExpr() : new ArrayList<>();
+ List<Expr> isNullExpr = analyzer.getOnIsNullExpr();
// isNullDeDuplication: De-Duplication for isNullExpr
Set<Expr> isNullDeDuplication =
(clauseType == ExprRewriter.ClauseType.ON_CLAUSE) ? analyzer.getOnIsNullDeDuplication() : Sets.newHashSet();
// inExpr: Record existing and infer in predicate
- List<Expr> inExpr =
- (clauseType == ExprRewriter.ClauseType.ON_CLAUSE) ? analyzer.getInExpr() : new ArrayList<>();
+ List<Expr> inExpr = analyzer.getInExpr();
// inDeDuplication: De-Duplication for inExpr
Set<Expr> inDeDuplication =
diff --git a/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java b/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java
index 15234fb5c3..c3c14f9ca5 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/rewrite/InferFiltersRuleTest.java
@@ -299,4 +299,25 @@ public class InferFiltersRuleTest {
Assert.assertTrue(planString.contains("`tb2`.`k1` >= 1"));
Assert.assertTrue(planString.contains("`tb2`.`k1` <= 2"));
}
+
+ @Test
+ public void testOnAndWhere2TablesLeftJoin() throws Exception {
+ SessionVariable sessionVariable = dorisAssert.getSessionVariable();
+ sessionVariable.setEnableInferPredicate(true);
+ Assert.assertTrue(sessionVariable.isEnableInferPredicate());
+ String query = "select * from tb1 left join tb2 on tb1.k1 = tb2.k1 where tb2.k1 = 1";
+ String planString = dorisAssert.query(query).explainQuery();
+ Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
+ }
+
+ @Test
+ public void testOnAndWhere2TablesInnerJoin() throws Exception {
+ SessionVariable sessionVariable = dorisAssert.getSessionVariable();
+ sessionVariable.setEnableInferPredicate(true);
+ Assert.assertTrue(sessionVariable.isEnableInferPredicate());
+ String query = "select * from tb1 inner join tb2 on tb1.k1 = tb2.k1 where tb2.k1 = 1";
+ String planString = dorisAssert.query(query).explainQuery();
+ Assert.assertTrue(planString.contains("`tb1`.`k1` = 1"));
+ }
+
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 02/05: [Enhancement] [Vectorized] Refactor and optimize BinaryOperation (#9087)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit af5ff046d1604a29470c4adb3d85d6c49d469328
Author: Pxl <95...@qq.com>
AuthorDate: Sat May 7 10:55:15 2022 +0800
[Enhancement] [Vectorized] Refactor and optimize BinaryOperation (#9087)
---
be/src/vec/data_types/data_type_decimal.h | 6 +-
be/src/vec/data_types/number_traits.h | 89 +--
be/src/vec/functions/divide.cpp | 34 +-
be/src/vec/functions/function.h | 9 -
be/src/vec/functions/function_binary_arithmetic.h | 881 ++++++++++++---------
.../function_binary_arithmetic_to_null_type.h | 247 ------
be/src/vec/functions/function_bit.cpp | 8 +-
be/src/vec/functions/function_cast.h | 1 -
be/src/vec/functions/int_div.cpp | 118 +--
be/src/vec/functions/int_div.h | 49 +-
be/src/vec/functions/math.cpp | 39 +-
be/src/vec/functions/minus.cpp | 2 +-
be/src/vec/functions/modulo.cpp | 180 ++---
be/src/vec/functions/multiply.cpp | 2 +-
be/src/vec/functions/plus.cpp | 2 +-
15 files changed, 677 insertions(+), 990 deletions(-)
diff --git a/be/src/vec/data_types/data_type_decimal.h b/be/src/vec/data_types/data_type_decimal.h
index d701ecee71..0613653324 100644
--- a/be/src/vec/data_types/data_type_decimal.h
+++ b/be/src/vec/data_types/data_type_decimal.h
@@ -106,10 +106,12 @@ public:
}
// Now, Doris only support precision:27, scale: 9
- DCHECK(precision_ == 27);
- DCHECK(scale_ == 9);
+ DCHECK(precision == 27);
+ DCHECK(scale == 9);
}
+ DataTypeDecimal(const DataTypeDecimal& rhs) : precision(rhs.precision), scale(rhs.scale) {}
+
const char* get_family_name() const override { return "Decimal"; }
std::string do_get_name() const override;
TypeIndex get_type_id() const override { return TypeId<T>::value; }
diff --git a/be/src/vec/data_types/number_traits.h b/be/src/vec/data_types/number_traits.h
index 8405215a05..8b87e55d93 100644
--- a/be/src/vec/data_types/number_traits.h
+++ b/be/src/vec/data_types/number_traits.h
@@ -22,6 +22,8 @@
#include <type_traits>
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_vector.h"
#include "vec/common/uint128.h"
#include "vec/core/types.h"
@@ -155,7 +157,8 @@ struct ResultOfSubtraction {
*/
template <typename A, typename B>
struct ResultOfFloatingPointDivision {
- using Type = Float64;
+ using Type = std::conditional_t<IsDecimalNumber<A>, A,
+ std::conditional_t<IsDecimalNumber<B>, B, Float64>>;
};
/** For integer division, we get a number with the same number of bits as in divisible.
@@ -170,14 +173,9 @@ struct ResultOfIntegerDivision {
*/
template <typename A, typename B>
struct ResultOfModulo {
- using Type =
- typename Construct<std::is_signed_v<A> || std::is_signed_v<B>, std::is_floating_point_v<A>, max(sizeof(A), sizeof(B))>::Type;
-};
-
-template <typename A>
-struct ResultOfNegate {
- using Type = typename Construct<true, std::is_floating_point_v<A>,
- std::is_signed_v<A> ? sizeof(A) : next_size(sizeof(A))>::Type;
+ using Type = typename Construct<std::is_signed_v<A> || std::is_signed_v<B>,
+ std::is_floating_point_v<A> || std::is_floating_point_v<B>,
+ max(sizeof(A), sizeof(B))>::Type;
};
template <typename A>
@@ -200,76 +198,15 @@ struct ResultOfBitNot {
using Type = typename Construct<std::is_signed_v<A>, false, sizeof(A)>::Type;
};
-/** Type casting for `if` function:
- * UInt<x>, UInt<y> -> UInt<max(x,y)>
- * Int<x>, Int<y> -> Int<max(x,y)>
- * Float<x>, Float<y> -> Float<max(x, y)>
- * UInt<x>, Int<y> -> Int<max(x*2, y)>
- * Float<x>, [U]Int<y> -> Float<max(x, y*2)>
- * Decimal<x>, Decimal<y> -> Decimal<max(x,y)>
- * UUID, UUID -> UUID
- * UInt64 , Int<x> -> Error
- * Float<x>, [U]Int64 -> Error
- */
template <typename A, typename B>
-struct ResultOfIf {
- static constexpr bool has_float = std::is_floating_point_v<A> || std::is_floating_point_v<B>;
- static constexpr bool has_integer = std::is_integral_v<A> || std::is_integral_v<B>;
- static constexpr bool has_signed = std::is_signed_v<A> || std::is_signed_v<B>;
- static constexpr bool has_unsigned = !std::is_signed_v<A> || !std::is_signed_v<B>;
-
- static constexpr size_t max_size_of_unsigned_integer =
- max(std::is_signed_v<A> ? 0 : sizeof(A), std::is_signed_v<B> ? 0 : sizeof(B));
- static constexpr size_t max_size_of_signed_integer =
- max(std::is_signed_v<A> ? sizeof(A) : 0, std::is_signed_v<B> ? sizeof(B) : 0);
- static constexpr size_t max_size_of_integer =
- max(std::is_integral_v<A> ? sizeof(A) : 0, std::is_integral_v<B> ? sizeof(B) : 0);
- static constexpr size_t max_size_of_float = max(std::is_floating_point_v<A> ? sizeof(A) : 0,
- std::is_floating_point_v<B> ? sizeof(B) : 0);
-
- using ConstructedType =
- typename Construct<has_signed, has_float,
- ((has_float && has_integer &&
- max_size_of_integer >= max_size_of_float) ||
- (has_signed && has_unsigned &&
- max_size_of_unsigned_integer >= max_size_of_signed_integer))
- ? max(sizeof(A), sizeof(B)) * 2
- : max(sizeof(A), sizeof(B))>::Type;
-
- using ConstructedWithUUID =
- std::conditional_t<std::is_same_v<A, UInt128> && std::is_same_v<B, UInt128>, A,
- ConstructedType>;
-
- using Type = std::conditional_t<
- !IsDecimalNumber<A> && !IsDecimalNumber<B>, ConstructedWithUUID,
- std::conditional_t<IsDecimalNumber<A> && IsDecimalNumber<B>,
- std::conditional_t<(sizeof(A) > sizeof(B)), A, B>, Error>>;
+struct BinaryOperatorTraits {
+ using ColumnVectorA = std::conditional_t<IsDecimalNumber<A>, ColumnDecimal<A>, ColumnVector<A>>;
+ using ColumnVectorB = std::conditional_t<IsDecimalNumber<B>, ColumnDecimal<B>, ColumnVector<B>>;
+ using ArrayA = typename ColumnVectorA::Container;
+ using ArrayB = typename ColumnVectorB::Container;
+ using ArrayNull = PaddedPODArray<UInt8>;
};
-/** Before applying operator `%` and bitwise operations, operands are casted to whole numbers. */
-template <typename A>
-struct ToInteger {
- using Type = typename Construct<std::is_signed_v<A>, false,
- std::is_floating_point_v<A> ? 8 : sizeof(A)>::Type;
-};
-
-// CLICKHOUSE-29. The same depth, different signs
-// NOTE: This case is applied for 64-bit integers only (for backward compatibility), but could be used for any-bit integers
-template <typename A, typename B>
-constexpr bool LeastGreatestSpecialCase = std::is_integral_v<A>&& std::is_integral_v<B> &&
- (8 == sizeof(A) && sizeof(A) == sizeof(B)) &&
- (std::is_signed_v<A> ^ std::is_signed_v<B>);
-
-template <typename A, typename B>
-using ResultOfLeast = std::conditional_t<LeastGreatestSpecialCase<A, B>,
- typename Construct<true, false, sizeof(A)>::Type,
- typename ResultOfIf<A, B>::Type>;
-
-template <typename A, typename B>
-using ResultOfGreatest = std::conditional_t<LeastGreatestSpecialCase<A, B>,
- typename Construct<false, false, sizeof(A)>::Type,
- typename ResultOfIf<A, B>::Type>;
-
} // namespace NumberTraits
} // namespace doris::vectorized
diff --git a/be/src/vec/functions/divide.cpp b/be/src/vec/functions/divide.cpp
index 71de120c98..a602cb8b33 100644
--- a/be/src/vec/functions/divide.cpp
+++ b/be/src/vec/functions/divide.cpp
@@ -18,8 +18,7 @@
// https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/divide.cpp
// and modified by Doris
-
-#include "vec/functions/function_binary_arithmetic_to_null_type.h"
+#include "vec/functions/function_binary_arithmetic.h"
#include "vec/functions/simple_function_factory.h"
namespace doris::vectorized {
@@ -29,25 +28,42 @@ static const DecimalV2Value one(1, 0);
template <typename A, typename B>
struct DivideFloatingImpl {
using ResultType = typename NumberTraits::ResultOfFloatingPointDivision<A, B>::Type;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
+
static const constexpr bool allow_decimal = true;
+ template <typename Result = ResultType>
+ static void apply(const typename Traits::ArrayA& a, B b,
+ typename ColumnVector<Result>::Container& c,
+ typename Traits::ArrayNull& null_map) {
+ size_t size = c.size();
+ UInt8 is_null = b == 0;
+ memset(null_map.data(), is_null, size);
+
+ if (!is_null) {
+ for (size_t i = 0; i < size; i++) {
+ c[i] = (double)a[i] / (double)b;
+ }
+ }
+ }
+
template <typename Result = DecimalV2Value>
- static inline DecimalV2Value apply(DecimalV2Value a, DecimalV2Value b, NullMap& null_map, size_t index) {
- null_map[index] = b.is_zero();
- return a / (b.is_zero() ? one : b);
+ static inline DecimalV2Value apply(DecimalV2Value a, DecimalV2Value b, UInt8& is_null) {
+ is_null = b.is_zero();
+ return a / (is_null ? one : b);
}
template <typename Result = ResultType>
- static inline Result apply(A a, B b, NullMap& null_map, size_t index) {
- null_map[index] = b == 0;
- return static_cast<Result>(a) / (b + (b == 0));
+ static inline Result apply(A a, B b, UInt8& is_null) {
+ is_null = b == 0;
+ return static_cast<Result>(a) / (b + is_null);
}
};
struct NameDivide {
static constexpr auto name = "divide";
};
-using FunctionDivide = FunctionBinaryArithmeticToNullType<DivideFloatingImpl, NameDivide>;
+using FunctionDivide = FunctionBinaryArithmetic<DivideFloatingImpl, NameDivide, true>;
void register_function_divide(SimpleFunctionFactory& factory) {
factory.register_function<FunctionDivide>();
diff --git a/be/src/vec/functions/function.h b/be/src/vec/functions/function.h
index 0ea494a06b..802ef6fdf4 100644
--- a/be/src/vec/functions/function.h
+++ b/be/src/vec/functions/function.h
@@ -95,11 +95,6 @@ protected:
*/
virtual ColumnNumbers get_arguments_that_are_always_constant() const { return {}; }
- /** True if function can be called on default arguments (include Nullable's) and won't throw.
- * Counterexample: modulo(0, 0)
- */
- virtual bool can_be_executed_on_default_arguments() const { return true; }
-
private:
Status default_implementation_for_nulls(FunctionContext* context, Block& block,
const ColumnNumbers& args, size_t result,
@@ -386,7 +381,6 @@ public:
bool use_default_implementation_for_constants() const override { return false; }
bool use_default_implementation_for_low_cardinality_columns() const override { return true; }
ColumnNumbers get_arguments_that_are_always_constant() const override { return {}; }
- bool can_be_executed_on_default_arguments() const override { return true; }
bool can_be_executed_on_low_cardinality_dictionary() const override {
return is_deterministic_in_scope_of_query();
}
@@ -460,9 +454,6 @@ protected:
ColumnNumbers get_arguments_that_are_always_constant() const final {
return function->get_arguments_that_are_always_constant();
}
- bool can_be_executed_on_default_arguments() const override {
- return function->can_be_executed_on_default_arguments();
- }
private:
std::shared_ptr<IFunction> function;
diff --git a/be/src/vec/functions/function_binary_arithmetic.h b/be/src/vec/functions/function_binary_arithmetic.h
index dd36158646..74b7df4260 100644
--- a/be/src/vec/functions/function_binary_arithmetic.h
+++ b/be/src/vec/functions/function_binary_arithmetic.h
@@ -20,167 +20,210 @@
#pragma once
-#include "common/logging.h"
+#include "runtime/tuple.h"
#include "vec/columns/column_const.h"
#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
#include "vec/columns/column_vector.h"
-#include "vec/common/assert_cast.h"
-#include "vec/common/typeid_cast.h"
-#include "vec/data_types/data_type.h"
-#include "vec/data_types/data_type_decimal.h"
-#include "vec/data_types/data_type_number.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_nullable.h"
#include "vec/data_types/number_traits.h"
#include "vec/functions/cast_type_to_either.h"
#include "vec/functions/function.h"
-#include "vec/functions/function_helpers.h"
-#include "vec/functions/int_div.h"
-#include "vec/utils/util.hpp"
namespace doris::vectorized {
-/** Arithmetic operations: +, -, *,
- * Bitwise operations: |, &, ^, ~.
- * Etc.
- */
+// Arithmetic operations: +, -, *, |, &, ^, ~
+// need implement apply(a, b)
-template <typename A, typename B, typename Op, typename ResultType_ = typename Op::ResultType>
+// Arithmetic operations (to null type): /, %, intDiv (integer division), log
+// need implement apply(a, b, is_null), apply(array_a, b, null_map)
+// apply(array_a, b, null_map) is only used on vector_constant
+
+// TODO: vector_constant optimization not work on decimal type now
+
+template <typename, typename>
+struct PlusImpl;
+template <typename, typename>
+struct MinusImpl;
+template <typename, typename>
+struct MultiplyImpl;
+template <typename, typename>
+struct DivideFloatingImpl;
+template <typename, typename>
+struct DivideIntegralImpl;
+template <typename, typename>
+struct ModuloImpl;
+
+template <template <typename, typename> typename Operation>
+struct OperationTraits {
+ using T = UInt8;
+ static constexpr bool is_plus_minus = std::is_same_v<Operation<T, T>, PlusImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, MinusImpl<T, T>>;
+ static constexpr bool is_multiply = std::is_same_v<Operation<T, T>, MultiplyImpl<T, T>>;
+ static constexpr bool is_division = std::is_same_v<Operation<T, T>, DivideFloatingImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, DivideIntegralImpl<T, T>>;
+ static constexpr bool allow_decimal =
+ std::is_same_v<Operation<T, T>, PlusImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, MinusImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, MultiplyImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, ModuloImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, DivideFloatingImpl<T, T>> ||
+ std::is_same_v<Operation<T, T>, DivideIntegralImpl<T, T>>;
+ static constexpr bool can_overflow = is_plus_minus || is_multiply;
+};
+
+template <typename A, typename B, typename Op, typename ResultType = typename Op::ResultType>
struct BinaryOperationImplBase {
- using ResultType = ResultType_;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
+ using ColumnVectorResult =
+ std::conditional_t<IsDecimalNumber<ResultType>, ColumnDecimal<ResultType>,
+ ColumnVector<ResultType>>;
- static void NO_INLINE vector_vector(const PaddedPODArray<A>& a, const PaddedPODArray<B>& b,
- PaddedPODArray<ResultType>& c) {
+ static void vector_vector(const PaddedPODArray<A>& a, const PaddedPODArray<B>& b,
+ PaddedPODArray<ResultType>& c) {
size_t size = a.size();
for (size_t i = 0; i < size; ++i) {
c[i] = Op::template apply<ResultType>(a[i], b[i]);
}
}
- static void NO_INLINE vector_vector(const PaddedPODArray<A>& a, const PaddedPODArray<B>& b,
- PaddedPODArray<ResultType>& c, NullMap& null_map) {
+ static void vector_vector(const PaddedPODArray<A>& a, const PaddedPODArray<B>& b,
+ PaddedPODArray<ResultType>& c, PaddedPODArray<UInt8>& null_map) {
size_t size = a.size();
for (size_t i = 0; i < size; ++i) {
- c[i] = Op::template apply<ResultType>(a[i], b[i], null_map, i);
+ c[i] = Op::template apply<ResultType>(a[i], b[i], null_map[i]);
}
}
- static void NO_INLINE vector_constant(const PaddedPODArray<A>& a, B b,
- PaddedPODArray<ResultType>& c) {
+ static void vector_constant(const PaddedPODArray<A>& a, B b, PaddedPODArray<ResultType>& c) {
size_t size = a.size();
- for (size_t i = 0; i < size; ++i) c[i] = Op::template apply<ResultType>(a[i], b);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = Op::template apply<ResultType>(a[i], b);
+ }
}
- static void NO_INLINE constant_vector(A a, const PaddedPODArray<B>& b,
- PaddedPODArray<ResultType>& c) {
+ static void vector_constant(const PaddedPODArray<A>& a, B b, PaddedPODArray<ResultType>& c,
+ PaddedPODArray<UInt8>& null_map) {
+ Op::template apply<ResultType>(a, b, c, null_map);
+ }
+
+ static void constant_vector(A a, const PaddedPODArray<B>& b, PaddedPODArray<ResultType>& c) {
size_t size = b.size();
- for (size_t i = 0; i < size; ++i) c[i] = Op::template apply<ResultType>(a, b[i]);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = Op::template apply<ResultType>(a, b[i]);
+ }
+ }
+
+ static void constant_vector(A a, const PaddedPODArray<B>& b, PaddedPODArray<ResultType>& c,
+ PaddedPODArray<UInt8>& null_map) {
+ size_t size = b.size();
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = Op::template apply<ResultType>(a, b[i], null_map[i]);
+ }
}
static ResultType constant_constant(A a, B b) { return Op::template apply<ResultType>(a, b); }
+
+ static ResultType constant_constant(A a, B b, UInt8& is_null) {
+ return Op::template apply<ResultType>(a, b, is_null);
+ }
};
-template <typename A, typename B, typename Op, typename ResultType = typename Op::ResultType>
-struct BinaryOperationImpl : BinaryOperationImplBase<A, B, Op, ResultType> {};
+template <typename A, typename B, typename Op, bool is_to_null_type,
+ typename ResultType = typename Op::ResultType>
+struct BinaryOperationImpl {
+ using Base = BinaryOperationImplBase<A, B, Op, ResultType>;
+
+ static ColumnPtr adapt_normal_constant_constant(A a, B b) {
+ auto column_result = Base::ColumnVectorResult::create(1);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(1, 0);
+ column_result->get_element(0) = Base::constant_constant(a, b, null_map->get_element(0));
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ column_result->get_element(0) = Base::constant_constant(a, b);
+ return column_result;
+ }
+ }
-template <typename, typename>
-struct PlusImpl;
-template <typename, typename>
-struct MinusImpl;
-template <typename, typename>
-struct MultiplyImpl;
-template <typename, typename>
-struct DivideFloatingImpl;
-template <typename, typename>
-struct DivideIntegralImpl;
-template <typename, typename>
-struct DivideIntegralOrZeroImpl;
-template <typename, typename>
-struct LeastBaseImpl;
-template <typename, typename>
-struct GreatestBaseImpl;
-template <typename, typename>
-struct ModuloImpl;
+ static ColumnPtr adapt_normal_vector_constant(ColumnPtr column_left, B b) {
+ auto column_left_ptr =
+ check_and_get_column<typename Base::Traits::ColumnVectorA>(column_left);
+ auto column_result = Base::ColumnVectorResult::create(column_left->size());
+ DCHECK(column_left_ptr != nullptr);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(column_left->size(), 0);
+ Base::vector_constant(column_left_ptr->get_data(), b, column_result->get_data(),
+ null_map->get_data());
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ Base::vector_constant(column_left_ptr->get_data(), b, column_result->get_data());
+ return column_result;
+ }
+ }
+
+ static ColumnPtr adapt_normal_constant_vector(A a, ColumnPtr column_right) {
+ auto column_right_ptr =
+ check_and_get_column<typename Base::Traits::ColumnVectorB>(column_right);
+ auto column_result = Base::ColumnVectorResult::create(column_right->size());
+ DCHECK(column_right_ptr != nullptr);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(column_right->size(), 0);
+ Base::constant_vector(a, column_right_ptr->get_data(), column_result->get_data(),
+ null_map->get_data());
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ Base::constant_vector(a, column_right_ptr->get_data(), column_result->get_data());
+ return column_result;
+ }
+ }
+
+ static ColumnPtr adapt_normal_vector_vector(ColumnPtr column_left, ColumnPtr column_right) {
+ auto column_left_ptr =
+ check_and_get_column<typename Base::Traits::ColumnVectorA>(column_left);
+ auto column_right_ptr =
+ check_and_get_column<typename Base::Traits::ColumnVectorB>(column_right);
+
+ auto column_result = Base::ColumnVectorResult::create(column_left->size());
+ DCHECK(column_left_ptr != nullptr && column_right_ptr != nullptr);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(column_result->size(), 0);
+ Base::vector_vector(column_left_ptr->get_data(), column_right_ptr->get_data(),
+ column_result->get_data(), null_map->get_data());
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ Base::vector_vector(column_left_ptr->get_data(), column_right_ptr->get_data(),
+ column_result->get_data());
+ return column_result;
+ }
+ }
+};
/// Binary operations for Decimals need scale args
/// +|- scale one of args (which scale factor is not 1). ScaleR = oneof(Scale1, Scale2);
/// * no agrs scale. ScaleR = Scale1 + Scale2;
/// / first arg scale. ScaleR = Scale1 (scale_a = DecimalType<B>::get_scale()).
template <typename A, typename B, template <typename, typename> typename Operation,
- typename ResultType_, bool _check_overflow = true>
+ typename ResultType, bool is_to_null_type, bool check_overflow = false>
struct DecimalBinaryOperation {
- static constexpr bool is_plus_minus =
- std::is_same_v<Operation<Int32, Int32>, PlusImpl<Int32, Int32>> ||
- std::is_same_v<Operation<Int32, Int32>, MinusImpl<Int32, Int32>>;
- static constexpr bool is_multiply =
- std::is_same_v<Operation<Int32, Int32>, MultiplyImpl<Int32, Int32>>;
- static constexpr bool is_float_division =
- std::is_same_v<Operation<Int32, Int32>, DivideFloatingImpl<Int32, Int32>>;
- static constexpr bool is_int_division =
- std::is_same_v<Operation<Int32, Int32>, DivideIntegralImpl<Int32, Int32>> ||
- std::is_same_v<Operation<Int32, Int32>, DivideIntegralOrZeroImpl<Int32, Int32>>;
- static constexpr bool is_division = is_float_division || is_int_division;
- static constexpr bool is_compare =
- std::is_same_v<Operation<Int32, Int32>, LeastBaseImpl<Int32, Int32>> ||
- std::is_same_v<Operation<Int32, Int32>, GreatestBaseImpl<Int32, Int32>>;
- static constexpr bool is_plus_minus_compare = is_plus_minus || is_compare;
- static constexpr bool can_overflow = is_plus_minus || is_multiply;
+ using OpTraits = OperationTraits<Operation>;
- using ResultType = ResultType_;
using NativeResultType = typename NativeType<ResultType>::Type;
using Op = Operation<NativeResultType, NativeResultType>;
- using ColVecA = std::conditional_t<IsDecimalNumber<A>, ColumnDecimal<A>, ColumnVector<A>>;
- using ColVecB = std::conditional_t<IsDecimalNumber<B>, ColumnDecimal<B>, ColumnVector<B>>;
- using ArrayA = typename ColVecA::Container;
- using ArrayB = typename ColVecB::Container;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
using ArrayC = typename ColumnDecimal<ResultType>::Container;
- using SelfNoOverflow = DecimalBinaryOperation<A, B, Operation, ResultType_, false>;
- static void vector_vector(const ArrayA& a, const ArrayB& b, ArrayC& c, ResultType scale_a,
- ResultType scale_b, bool check_overflow) {
- if (check_overflow)
- vector_vector(a, b, c, scale_a, scale_b);
- else
- SelfNoOverflow::vector_vector(a, b, c, scale_a, scale_b);
- }
-
- /// null_map for divide and mod
- static void vector_vector(const ArrayA& a, const ArrayB& b, ArrayC& c, ResultType scale_a,
- ResultType scale_b, bool check_overflow, NullMap& null_map) {
- if (check_overflow)
- vector_vector(a, b, c, scale_a, scale_b, null_map);
- else
- SelfNoOverflow::vector_vector(a, b, c, scale_a, scale_b, null_map);
- }
-
- static void vector_constant(const ArrayA& a, B b, ArrayC& c, ResultType scale_a,
- ResultType scale_b, bool check_overflow) {
- if (check_overflow)
- vector_constant(a, b, c, scale_a, scale_b);
- else
- SelfNoOverflow::vector_constant(a, b, c, scale_a, scale_b);
- }
-
- static void constant_vector(A a, const ArrayB& b, ArrayC& c, ResultType scale_a,
- ResultType scale_b, bool check_overflow) {
- if (check_overflow)
- constant_vector(a, b, c, scale_a, scale_b);
- else
- SelfNoOverflow::constant_vector(a, b, c, scale_a, scale_b);
- }
-
- static ResultType constant_constant(A a, B b, ResultType scale_a, ResultType scale_b,
- bool check_overflow) {
- if (check_overflow)
- return constant_constant(a, b, scale_a, scale_b);
- else
- return SelfNoOverflow::constant_constant(a, b, scale_a, scale_b);
- }
-
- static void NO_INLINE vector_vector(const ArrayA& a, const ArrayB& b, ArrayC& c,
- ResultType scale_a [[maybe_unused]],
- ResultType scale_b [[maybe_unused]]) {
+ static void vector_vector(const typename Traits::ArrayA& a, const typename Traits::ArrayB& b,
+ ArrayC& c, ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]]) {
size_t size = a.size();
- if constexpr (is_plus_minus_compare) {
+ if constexpr (OpTraits::is_plus_minus) {
if (scale_a != 1) {
for (size_t i = 0; i < size; ++i) {
c[i] = apply_scaled<true>(a[i], b[i], scale_a);
@@ -201,71 +244,207 @@ struct DecimalBinaryOperation {
}
/// null_map for divide and mod
- static void NO_INLINE vector_vector(const ArrayA& a, const ArrayB& b, ArrayC& c,
- ResultType scale_a [[maybe_unused]],
- ResultType scale_b [[maybe_unused]], NullMap& null_map) {
+ static void vector_vector(const typename Traits::ArrayA& a, const typename Traits::ArrayB& b,
+ ArrayC& c, ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]], NullMap& null_map) {
size_t size = a.size();
/// default: use it if no return before
for (size_t i = 0; i < size; ++i) {
- c[i] = apply(a[i], b[i], null_map, i);
+ c[i] = apply(a[i], b[i], null_map[i]);
}
}
- static void NO_INLINE vector_constant(const ArrayA& a, B b, ArrayC& c,
- ResultType scale_a [[maybe_unused]],
- ResultType scale_b [[maybe_unused]]) {
+ static void vector_constant(const typename Traits::ArrayA& a, B b, ArrayC& c,
+ ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]]) {
size_t size = a.size();
- if constexpr (is_plus_minus_compare) {
+ if constexpr (OpTraits::is_plus_minus) {
if (scale_a != 1) {
- for (size_t i = 0; i < size; ++i) c[i] = apply_scaled<true>(a[i], b, scale_a);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled<true>(a[i], b, scale_a);
+ }
return;
} else if (scale_b != 1) {
- for (size_t i = 0; i < size; ++i) c[i] = apply_scaled<false>(a[i], b, scale_b);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled<false>(a[i], b, scale_b);
+ }
return;
}
- } else if constexpr (is_division && IsDecimalNumber<B>) {
- for (size_t i = 0; i < size; ++i) c[i] = apply_scaled_div(a[i], b, scale_a);
+ } else if constexpr (OpTraits::is_division && IsDecimalNumber<B>) {
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled_div(a[i], b, scale_a);
+ }
return;
}
/// default: use it if no return before
- for (size_t i = 0; i < size; ++i) c[i] = apply(a[i], b);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply(a[i], b);
+ }
+ }
+
+ static void vector_constant(const typename Traits::ArrayA& a, B b, ArrayC& c,
+ ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]], NullMap& null_map) {
+ size_t size = a.size();
+ if constexpr (OpTraits::is_division && IsDecimalNumber<B>) {
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled_div(a[i], b, scale_a, null_map[i]);
+ }
+ return;
+ }
+
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply(a[i], b, null_map[i]);
+ }
}
- static void NO_INLINE constant_vector(A a, const ArrayB& b, ArrayC& c,
- ResultType scale_a [[maybe_unused]],
- ResultType scale_b [[maybe_unused]]) {
+ static void constant_vector(A a, const typename Traits::ArrayB& b, ArrayC& c,
+ ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]]) {
size_t size = b.size();
- if constexpr (is_plus_minus_compare) {
+ if constexpr (OpTraits::is_plus_minus) {
if (scale_a != 1) {
- for (size_t i = 0; i < size; ++i) c[i] = apply_scaled<true>(a, b[i], scale_a);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled<true>(a, b[i], scale_a);
+ }
return;
} else if (scale_b != 1) {
- for (size_t i = 0; i < size; ++i) c[i] = apply_scaled<false>(a, b[i], scale_b);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled<false>(a, b[i], scale_b);
+ }
return;
}
- } else if constexpr (is_division && IsDecimalNumber<B>) {
- for (size_t i = 0; i < size; ++i) c[i] = apply_scaled_div(a, b[i], scale_a);
+ } else if constexpr (OpTraits::is_division && IsDecimalNumber<B>) {
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled_div(a, b[i], scale_a);
+ }
return;
}
/// default: use it if no return before
- for (size_t i = 0; i < size; ++i) c[i] = apply(a, b[i]);
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply(a, b[i]);
+ }
+ }
+
+ static void constant_vector(A a, const typename Traits::ArrayB& b, ArrayC& c,
+ ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]], NullMap& null_map) {
+ size_t size = b.size();
+ if constexpr (OpTraits::is_division && IsDecimalNumber<B>) {
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply_scaled_div(a, b[i], scale_a, null_map[i]);
+ }
+ return;
+ }
+
+ for (size_t i = 0; i < size; ++i) {
+ c[i] = apply(a, b[i], null_map[i]);
+ }
}
static ResultType constant_constant(A a, B b, ResultType scale_a [[maybe_unused]],
ResultType scale_b [[maybe_unused]]) {
- if constexpr (is_plus_minus_compare) {
- if (scale_a != 1)
+ if constexpr (OpTraits::is_plus_minus) {
+ if (scale_a != 1) {
return apply_scaled<true>(a, b, scale_a);
- else if (scale_b != 1)
+ } else if (scale_b != 1) {
return apply_scaled<false>(a, b, scale_b);
- } else if constexpr (is_division && IsDecimalNumber<B>)
+ }
+ } else if constexpr (OpTraits::is_division && IsDecimalNumber<B>) {
return apply_scaled_div(a, b, scale_a);
+ }
return apply(a, b);
}
+ static ResultType constant_constant(A a, B b, ResultType scale_a [[maybe_unused]],
+ ResultType scale_b [[maybe_unused]], UInt8& is_null) {
+ if constexpr (OpTraits::is_plus_minus) {
+ if (scale_a != 1) {
+ return apply_scaled<true>(a, b, scale_a, is_null);
+ } else if (scale_b != 1) {
+ return apply_scaled<false>(a, b, scale_b, is_null);
+ }
+ } else if constexpr (OpTraits::is_division && IsDecimalNumber<B>) {
+ return apply_scaled_div(a, b, scale_a, is_null);
+ }
+ return apply(a, b, is_null);
+ }
+
+ static ColumnPtr adapt_decimal_constant_constant(A a, B b, UInt32 scale, ResultType scale_a,
+ ResultType scale_b) {
+ auto column_result = ColumnDecimal<ResultType>::create(1, scale);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(1, 0);
+ column_result->get_element(0) =
+ constant_constant(a, b, scale_a, scale_b, null_map->get_element(0));
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ column_result->get_element(0) = constant_constant(a, b, scale_a, scale_b);
+ return column_result;
+ }
+ }
+
+ static ColumnPtr adapt_decimal_vector_constant(ColumnPtr column_left, B b, UInt32 column_scale,
+ ResultType scale_a, ResultType scale_b) {
+ auto column_left_ptr = check_and_get_column<typename Traits::ColumnVectorA>(column_left);
+ auto column_result = ColumnDecimal<ResultType>::create(column_left->size(), column_scale);
+ DCHECK(column_left_ptr != nullptr);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(column_left->size(), 0);
+ vector_constant(column_left_ptr->get_data(), b, column_result->get_data(), scale_a,
+ scale_b, null_map->get_data());
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ vector_constant(column_left_ptr->get_data(), b, column_result->get_data(), scale_a,
+ scale_b);
+ return column_result;
+ }
+ }
+
+ static ColumnPtr adapt_decimal_constant_vector(A a, ColumnPtr column_right, UInt32 column_scale,
+ ResultType scale_a, ResultType scale_b) {
+ auto column_right_ptr = check_and_get_column<typename Traits::ColumnVectorB>(column_right);
+ auto column_result = ColumnDecimal<ResultType>::create(column_right->size(), column_scale);
+ DCHECK(column_right_ptr != nullptr);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(column_right->size(), 0);
+ constant_vector(a, column_right_ptr->get_data(), column_result->get_data(), scale_a,
+ scale_b, null_map->get_data());
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ constant_vector(a, column_right_ptr->get_data(), column_result->get_data(), scale_a,
+ scale_b);
+ return column_result;
+ }
+ }
+
+ static ColumnPtr adapt_decimal_vector_vector(ColumnPtr column_left, ColumnPtr column_right,
+ UInt32 column_scale, ResultType scale_a,
+ ResultType scale_b) {
+ auto column_left_ptr = check_and_get_column<typename Traits::ColumnVectorA>(column_left);
+ auto column_right_ptr = check_and_get_column<typename Traits::ColumnVectorB>(column_right);
+
+ auto column_result = ColumnDecimal<ResultType>::create(column_left->size(), column_scale);
+ DCHECK(column_left_ptr != nullptr && column_right_ptr != nullptr);
+
+ if constexpr (is_to_null_type) {
+ auto null_map = ColumnUInt8::create(column_result->size(), 0);
+ vector_vector(column_left_ptr->get_data(), column_right_ptr->get_data(),
+ column_result->get_data(), scale_a, scale_b, null_map->get_data());
+ return ColumnNullable::create(std::move(column_result), std::move(null_map));
+ } else {
+ vector_vector(column_left_ptr->get_data(), column_right_ptr->get_data(),
+ column_result->get_data(), scale_a, scale_b);
+ return column_result;
+ }
+ }
+
private:
/// there's implicit type convertion here
static NativeResultType apply(NativeResultType a, NativeResultType b) {
@@ -280,11 +459,10 @@ private:
}
/// null_map for divide and mod
- static NativeResultType apply(NativeResultType a, NativeResultType b, NullMap& null_map,
- size_t index) {
+ static NativeResultType apply(NativeResultType a, NativeResultType b, UInt8& is_null) {
DecimalV2Value l(a);
DecimalV2Value r(b);
- auto ans = Op::template apply(l, r, null_map, index);
+ auto ans = Op::template apply(l, r, is_null);
NativeResultType result;
memcpy(&result, &ans, std::min(sizeof(result), sizeof(ans)));
return result;
@@ -293,30 +471,33 @@ private:
template <bool scale_left>
static NativeResultType apply_scaled(NativeResultType a, NativeResultType b,
NativeResultType scale) {
- if constexpr (is_plus_minus_compare) {
+ if constexpr (OpTraits::is_plus_minus) {
NativeResultType res;
- if constexpr (_check_overflow) {
+ if constexpr (check_overflow) {
bool overflow = false;
- if constexpr (scale_left)
+ if constexpr (scale_left) {
overflow |= common::mul_overflow(a, scale, a);
- else
+ } else {
overflow |= common::mul_overflow(b, scale, b);
+ }
- if constexpr (can_overflow)
+ if constexpr (OpTraits::can_overflow) {
overflow |= Op::template apply<NativeResultType>(a, b, res);
- else
+ } else {
res = Op::template apply<NativeResultType>(a, b);
+ }
if (overflow) {
LOG(FATAL) << "Decimal math overflow";
}
} else {
- if constexpr (scale_left)
+ if constexpr (scale_left) {
a *= scale;
- else
+ } else {
b *= scale;
- res = Op::template apply<NativeResultType>(a, b);
+ }
+ res = apply(a, b);
}
return res;
@@ -324,23 +505,25 @@ private:
}
static NativeResultType apply_scaled_div(NativeResultType a, NativeResultType b,
- NativeResultType scale, NullMap& null_map,
- size_t index) {
- if constexpr (is_division) {
- if constexpr (_check_overflow) {
+ NativeResultType scale, UInt8& is_null) {
+ if constexpr (OpTraits::is_division) {
+ if constexpr (check_overflow) {
bool overflow = false;
- if constexpr (!IsDecimalNumber<A>)
+ if constexpr (!IsDecimalNumber<A>) {
overflow |= common::mul_overflow(scale, scale, scale);
+ }
overflow |= common::mul_overflow(a, scale, a);
if (overflow) {
LOG(FATAL) << "Decimal math overflow";
}
} else {
- if constexpr (!IsDecimalNumber<A>) scale *= scale;
+ if constexpr (!IsDecimalNumber<A>) {
+ scale *= scale;
+ }
a *= scale;
}
- return Op::template apply<NativeResultType>(a, b, null_map, index);
+ return apply(a, b, is_null);
}
}
};
@@ -362,12 +545,6 @@ constexpr bool IsIntegral = false;
template <>
inline constexpr bool IsIntegral<DataTypeUInt8> = true;
template <>
-inline constexpr bool IsIntegral<DataTypeUInt16> = true;
-template <>
-inline constexpr bool IsIntegral<DataTypeUInt32> = true;
-template <>
-inline constexpr bool IsIntegral<DataTypeUInt64> = true;
-template <>
inline constexpr bool IsIntegral<DataTypeInt8> = true;
template <>
inline constexpr bool IsIntegral<DataTypeInt16> = true;
@@ -378,14 +555,7 @@ inline constexpr bool IsIntegral<DataTypeInt64> = true;
template <>
inline constexpr bool IsIntegral<DataTypeInt128> = true;
-template <typename DataType>
-constexpr bool IsFloatingPoint = false;
-template <>
-inline constexpr bool IsFloatingPoint<DataTypeFloat32> = true;
-template <>
-inline constexpr bool IsFloatingPoint<DataTypeFloat64> = true;
-
-template <typename T0, typename T1>
+template <typename A, typename B>
constexpr bool UseLeftDecimal = false;
template <>
inline constexpr bool UseLeftDecimal<DataTypeDecimal<Decimal128>, DataTypeDecimal<Decimal32>> =
@@ -403,29 +573,13 @@ using DataTypeFromFieldType =
template <template <typename, typename> class Operation, typename LeftDataType,
typename RightDataType>
struct BinaryOperationTraits {
- using T0 = typename LeftDataType::FieldType;
- using T1 = typename RightDataType::FieldType;
-
-private: /// it's not correct for Decimal
- using Op = Operation<T0, T1>;
-
-public:
- static constexpr bool allow_decimal =
- std::is_same_v<Operation<T0, T0>, PlusImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, MinusImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, MultiplyImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, ModuloImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, DivideFloatingImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, DivideIntegralImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, DivideIntegralOrZeroImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, LeastBaseImpl<T0, T0>> ||
- std::is_same_v<Operation<T0, T0>, GreatestBaseImpl<T0, T0>>;
-
+ using Op = Operation<typename LeftDataType::FieldType, typename RightDataType::FieldType>;
+ using OpTraits = OperationTraits<Operation>;
/// Appropriate result type for binary operator on numeric types. "Date" can also mean
/// DateTime, but if both operands are Dates, their type must be the same (e.g. Date - DateTime is invalid).
using ResultDataType = Switch<
/// Decimal cases
- Case<!allow_decimal &&
+ Case<!OpTraits::allow_decimal &&
(IsDataTypeDecimal<LeftDataType> || IsDataTypeDecimal<RightDataType>),
InvalidType>,
Case<IsDataTypeDecimal<LeftDataType> && IsDataTypeDecimal<RightDataType> &&
@@ -451,20 +605,135 @@ public:
DataTypeFromFieldType<typename Op::ResultType>>>;
};
-template <template <typename, typename> class Op, typename Name,
- bool CanBeExecutedOnDefaultArguments = true>
+template <typename LeftDataType, typename RightDataType,
+ template <typename, typename> class Operation, bool is_to_null_type>
+struct ConstOrVectorAdapter {
+ static constexpr bool result_is_decimal =
+ IsDataTypeDecimal<LeftDataType> || IsDataTypeDecimal<RightDataType>;
+ using OpTraits = OperationTraits<Operation>;
+
+ using ResultDataType =
+ typename BinaryOperationTraits<Operation, LeftDataType, RightDataType>::ResultDataType;
+ using ResultType = typename ResultDataType::FieldType;
+ using A = typename LeftDataType::FieldType;
+ using B = typename RightDataType::FieldType;
+
+ using OperationImpl = std::conditional_t<
+ IsDataTypeDecimal<ResultDataType>,
+ DecimalBinaryOperation<A, B, Operation, ResultType, is_to_null_type>,
+ BinaryOperationImpl<A, B, Operation<A, B>, is_to_null_type, ResultType>>;
+
+ static ColumnPtr execute(ColumnPtr column_left, ColumnPtr column_right,
+ const LeftDataType& type_left, const RightDataType& type_right) {
+ bool is_const_left = is_column_const(*column_left);
+ bool is_const_right = is_column_const(*column_right);
+
+ if (is_const_left && is_const_right) {
+ return constant_constant(column_left, column_right, type_left, type_right);
+ } else if (is_const_left) {
+ return constant_vector(column_left, column_right, type_left, type_right);
+ } else if (is_const_right) {
+ return vector_constant(column_left, column_right, type_left, type_right);
+ } else {
+ return vector_vector(column_left, column_right, type_left, type_right);
+ }
+ }
+
+private:
+ static auto get_decimal_infos(const LeftDataType& type_left, const RightDataType& type_right) {
+ ResultDataType type = decimal_result_type(type_left, type_right, OpTraits::is_multiply,
+ OpTraits::is_division);
+ typename ResultDataType::FieldType scale_a =
+ type.scale_factor_for(type_left, OpTraits::is_multiply);
+ typename ResultDataType::FieldType scale_b =
+ type.scale_factor_for(type_right, OpTraits::is_multiply || OpTraits::is_division);
+ return std::make_tuple(type, scale_a, scale_b);
+ }
+
+ static ColumnPtr constant_constant(ColumnPtr column_left, ColumnPtr column_right,
+ const LeftDataType& type_left,
+ const RightDataType& type_right) {
+ auto column_left_ptr = check_and_get_column<ColumnConst>(column_left);
+ auto column_right_ptr = check_and_get_column<ColumnConst>(column_right);
+ DCHECK(column_left_ptr != nullptr && column_right_ptr != nullptr);
+
+ ColumnPtr column_result = nullptr;
+
+ if constexpr (result_is_decimal) {
+ auto [type, scale_a, scale_b] = get_decimal_infos(type_left, type_right);
+
+ column_result = OperationImpl::adapt_decimal_constant_constant(
+ column_left_ptr->template get_value<A>(),
+ column_right_ptr->template get_value<B>(), type.get_scale(), scale_a, scale_b);
+
+ } else {
+ column_result = OperationImpl::adapt_normal_constant_constant(
+ column_left_ptr->template get_value<A>(),
+ column_right_ptr->template get_value<B>());
+ }
+
+ return ColumnConst::create(std::move(column_result), column_left->size());
+ }
+
+ static ColumnPtr vector_constant(ColumnPtr column_left, ColumnPtr column_right,
+ const LeftDataType& type_left,
+ const RightDataType& type_right) {
+ auto column_right_ptr = check_and_get_column<ColumnConst>(column_right);
+ DCHECK(column_right_ptr != nullptr);
+
+ if constexpr (result_is_decimal) {
+ auto [type, scale_a, scale_b] = get_decimal_infos(type_left, type_right);
+
+ return OperationImpl::adapt_decimal_vector_constant(
+ column_left->get_ptr(), column_right_ptr->template get_value<B>(),
+ type.get_scale(), scale_a, scale_b);
+ } else {
+ return OperationImpl::adapt_normal_vector_constant(
+ column_left->get_ptr(), column_right_ptr->template get_value<B>());
+ }
+ }
+
+ static ColumnPtr constant_vector(ColumnPtr column_left, ColumnPtr column_right,
+ const LeftDataType& type_left,
+ const RightDataType& type_right) {
+ auto column_left_ptr = check_and_get_column<ColumnConst>(column_left);
+ DCHECK(column_left_ptr != nullptr);
+
+ if constexpr (result_is_decimal) {
+ auto [type, scale_a, scale_b] = get_decimal_infos(type_left, type_right);
+
+ return OperationImpl::adapt_decimal_constant_vector(
+ column_left_ptr->template get_value<A>(), column_right->get_ptr(),
+ type.get_scale(), scale_a, scale_b);
+ } else {
+ return OperationImpl::adapt_normal_constant_vector(
+ column_left_ptr->template get_value<A>(), column_right->get_ptr());
+ }
+ }
+
+ static ColumnPtr vector_vector(ColumnPtr column_left, ColumnPtr column_right,
+ const LeftDataType& type_left, const RightDataType& type_right) {
+ if constexpr (result_is_decimal) {
+ auto [type, scale_a, scale_b] = get_decimal_infos(type_left, type_right);
+
+ return OperationImpl::adapt_decimal_vector_vector(column_left->get_ptr(),
+ column_right->get_ptr(),
+ type.get_scale(), scale_a, scale_b);
+ } else {
+ return OperationImpl::adapt_normal_vector_vector(column_left->get_ptr(),
+ column_right->get_ptr());
+ }
+ }
+};
+
+template <template <typename, typename> class Operation, typename Name, bool is_to_null_type>
class FunctionBinaryArithmetic : public IFunction {
- bool check_decimal_overflow = true;
- static constexpr bool has_variadic_argument =
- !std::is_void_v<decltype(has_variadic_argument_types(std::declval<Op<int,int>>()))>;
+ using OpTraits = OperationTraits<Operation>;
template <typename F>
static bool cast_type(const IDataType* type, F&& f) {
- return cast_type_to_either<DataTypeUInt8, DataTypeUInt16, DataTypeUInt32, DataTypeUInt64,
- DataTypeInt8, DataTypeInt16, DataTypeInt32, DataTypeInt64,
- DataTypeInt128, DataTypeFloat32, DataTypeFloat64,
- // DataTypeDate,
- // DataTypeDateTime,
+ return cast_type_to_either<DataTypeUInt8, DataTypeInt8, DataTypeInt16, DataTypeInt32,
+ DataTypeInt64, DataTypeInt128, DataTypeFloat32, DataTypeFloat64,
DataTypeDecimal<Decimal32>, DataTypeDecimal<Decimal64>,
DataTypeDecimal<Decimal128>>(type, std::forward<F>(f));
}
@@ -476,39 +745,17 @@ class FunctionBinaryArithmetic : public IFunction {
});
}
- bool is_aggregate_multiply(const DataTypePtr& type0, const DataTypePtr& type1) const {
- if constexpr (!std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>) return false;
-
- WhichDataType which0(type0);
- WhichDataType which1(type1);
-
- return (which0.is_aggregate_function() && which1.is_native_uint()) ||
- (which0.is_native_uint() && which1.is_aggregate_function());
- }
-
- bool is_aggregate_addition(const DataTypePtr& type0, const DataTypePtr& type1) const {
- if constexpr (!std::is_same_v<Op<UInt8, UInt8>, PlusImpl<UInt8, UInt8>>) return false;
-
- WhichDataType which0(type0);
- WhichDataType which1(type1);
-
- return which0.is_aggregate_function() && which1.is_aggregate_function();
- }
-
public:
static constexpr auto name = Name::name;
+
static FunctionPtr create() { return std::make_shared<FunctionBinaryArithmetic>(); }
- FunctionBinaryArithmetic() {}
+ FunctionBinaryArithmetic() = default;
+
String get_name() const override { return name; }
size_t get_number_of_arguments() const override { return 2; }
- DataTypes get_variadic_argument_types_impl() const override {
- if constexpr (has_variadic_argument) return Op<int, int>::get_variadic_argument_types();
- return {};
- }
-
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
DataTypePtr type_res;
bool valid = cast_both_types(
@@ -516,29 +763,26 @@ public:
using LeftDataType = std::decay_t<decltype(left)>;
using RightDataType = std::decay_t<decltype(right)>;
using ResultDataType =
- typename BinaryOperationTraits<Op, LeftDataType,
+ typename BinaryOperationTraits<Operation, LeftDataType,
RightDataType>::ResultDataType;
if constexpr (!std::is_same_v<ResultDataType, InvalidType>) {
if constexpr (IsDataTypeDecimal<LeftDataType> &&
IsDataTypeDecimal<RightDataType>) {
- constexpr bool is_multiply =
- std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>;
- constexpr bool is_division = false;
-
- ResultDataType result_type =
- decimal_result_type(left, right, is_multiply, is_division);
+ ResultDataType result_type = decimal_result_type(
+ left, right, OpTraits::is_multiply, OpTraits::is_division);
type_res = std::make_shared<ResultDataType>(result_type.get_precision(),
result_type.get_scale());
- } else if constexpr (IsDataTypeDecimal<LeftDataType>)
+ } else if constexpr (IsDataTypeDecimal<LeftDataType>) {
type_res = std::make_shared<LeftDataType>(left.get_precision(),
left.get_scale());
- else if constexpr (IsDataTypeDecimal<RightDataType>)
+ } else if constexpr (IsDataTypeDecimal<RightDataType>) {
type_res = std::make_shared<RightDataType>(right.get_precision(),
right.get_scale());
- else if constexpr (IsDataTypeDecimal<ResultDataType>)
+ } else if constexpr (IsDataTypeDecimal<ResultDataType>) {
type_res = std::make_shared<ResultDataType>(27, 9);
- else
+ } else {
type_res = std::make_shared<ResultDataType>();
+ }
return true;
}
return false;
@@ -549,6 +793,10 @@ public:
get_name());
}
+ if constexpr (is_to_null_type) {
+ return make_nullable(type_res);
+ }
+
return type_res;
}
@@ -556,150 +804,29 @@ public:
size_t result, size_t input_rows_count) override {
auto* left_generic = block.get_by_position(arguments[0]).type.get();
auto* right_generic = block.get_by_position(arguments[1]).type.get();
+ if (left_generic->is_nullable()) {
+ left_generic =
+ static_cast<const DataTypeNullable*>(left_generic)->get_nested_type().get();
+ }
+ if (right_generic->is_nullable()) {
+ right_generic =
+ static_cast<const DataTypeNullable*>(right_generic)->get_nested_type().get();
+ }
+
bool valid = cast_both_types(
left_generic, right_generic, [&](const auto& left, const auto& right) {
using LeftDataType = std::decay_t<decltype(left)>;
using RightDataType = std::decay_t<decltype(right)>;
using ResultDataType =
- typename BinaryOperationTraits<Op, LeftDataType,
+ typename BinaryOperationTraits<Operation, LeftDataType,
RightDataType>::ResultDataType;
- if constexpr (!std::is_same_v<ResultDataType, InvalidType>) {
- constexpr bool result_is_decimal =
- IsDataTypeDecimal<LeftDataType> || IsDataTypeDecimal<RightDataType>;
- constexpr bool is_multiply =
- std::is_same_v<Op<UInt8, UInt8>, MultiplyImpl<UInt8, UInt8>>;
- constexpr bool is_division = false;
-
- using T0 = typename LeftDataType::FieldType;
- using T1 = typename RightDataType::FieldType;
- using ResultType = typename ResultDataType::FieldType;
- using ColVecT0 = std::conditional_t<IsDecimalNumber<T0>, ColumnDecimal<T0>,
- ColumnVector<T0>>;
- using ColVecT1 = std::conditional_t<IsDecimalNumber<T1>, ColumnDecimal<T1>,
- ColumnVector<T1>>;
- using ColVecResult = std::conditional_t<IsDecimalNumber<ResultType>,
- ColumnDecimal<ResultType>,
- ColumnVector<ResultType>>;
-
- /// Decimal operations need scale. Operations are on result type.
- using OpImpl = std::conditional_t<
- IsDataTypeDecimal<ResultDataType>,
- DecimalBinaryOperation<T0, T1, Op, ResultType>,
- BinaryOperationImpl<T0, T1, Op<T0, T1>, ResultType>>;
-
- auto col_left_raw = block.get_by_position(arguments[0]).column.get();
- auto col_right_raw = block.get_by_position(arguments[1]).column.get();
- if (auto col_left = check_and_get_column_const<ColVecT0>(col_left_raw)) {
- if (auto col_right =
- check_and_get_column_const<ColVecT1>(col_right_raw)) {
- /// the only case with a non-vector result
- if constexpr (result_is_decimal) {
- ResultDataType type = decimal_result_type(
- left, right, is_multiply, is_division);
- typename ResultDataType::FieldType scale_a =
- type.scale_factor_for(left, is_multiply);
- typename ResultDataType::FieldType scale_b =
- type.scale_factor_for(right,
- is_multiply || is_division);
-
- auto res = OpImpl::constant_constant(
- col_left->template get_value<T0>(),
- col_right->template get_value<T1>(), scale_a, scale_b,
- check_decimal_overflow);
- block.get_by_position(result).column =
- ResultDataType(type.get_precision(), type.get_scale())
- .create_column_const(
- col_left->size(),
- to_field(res, type.get_scale()));
-
- } else {
- auto res = OpImpl::constant_constant(
- col_left->template get_value<T0>(),
- col_right->template get_value<T1>());
- block.get_by_position(result).column =
- ResultDataType().create_column_const(col_left->size(),
- to_field(res));
- }
- return true;
- }
- }
- typename ColVecResult::MutablePtr col_res = nullptr;
- if constexpr (result_is_decimal) {
- ResultDataType type =
- decimal_result_type(left, right, is_multiply, is_division);
- col_res = ColVecResult::create(0, type.get_scale());
- } else
- col_res = ColVecResult::create();
-
- auto& vec_res = col_res->get_data();
- vec_res.resize(block.rows());
-
- if (auto col_left_const =
- check_and_get_column_const<ColVecT0>(col_left_raw)) {
- if (auto col_right = check_and_get_column<ColVecT1>(col_right_raw)) {
- if constexpr (result_is_decimal) {
- ResultDataType type = decimal_result_type(
- left, right, is_multiply, is_division);
-
- typename ResultDataType::FieldType scale_a =
- type.scale_factor_for(left, is_multiply);
- typename ResultDataType::FieldType scale_b =
- type.scale_factor_for(right,
- is_multiply || is_division);
- if constexpr (IsDataTypeDecimal<RightDataType> && is_division)
- scale_a = right.get_scale_multiplier();
-
- OpImpl::constant_vector(
- col_left_const->template get_value<T0>(),
- col_right->get_data(), vec_res, scale_a, scale_b,
- check_decimal_overflow);
- } else
- OpImpl::constant_vector(
- col_left_const->template get_value<T0>(),
- col_right->get_data(), vec_res);
- } else
- return false;
- } else if (auto col_left = check_and_get_column<ColVecT0>(col_left_raw)) {
- if constexpr (result_is_decimal) {
- ResultDataType type =
- decimal_result_type(left, right, is_multiply, is_division);
-
- typename ResultDataType::FieldType scale_a =
- type.scale_factor_for(left, is_multiply);
- typename ResultDataType::FieldType scale_b =
- type.scale_factor_for(right, is_multiply || is_division);
- if (auto col_right =
- check_and_get_column<ColVecT1>(col_right_raw)) {
- OpImpl::vector_vector(col_left->get_data(),
- col_right->get_data(), vec_res, scale_a,
- scale_b, check_decimal_overflow);
- } else if (auto col_right_const =
- check_and_get_column_const<ColVecT1>(
- col_right_raw)) {
- OpImpl::vector_constant(
- col_left->get_data(),
- col_right_const->template get_value<T1>(), vec_res,
- scale_a, scale_b, check_decimal_overflow);
- } else
- return false;
- } else {
- if (auto col_right = check_and_get_column<ColVecT1>(col_right_raw))
- OpImpl::vector_vector(col_left->get_data(),
- col_right->get_data(), vec_res);
- else if (auto col_right_const =
- check_and_get_column_const<ColVecT1>(
- col_right_raw))
- OpImpl::vector_constant(
- col_left->get_data(),
- col_right_const->template get_value<T1>(), vec_res);
- else
- return false;
- }
- } else
- return false;
-
- block.replace_by_position(result, std::move(col_res));
+ if constexpr (!std::is_same_v<ResultDataType, InvalidType>) {
+ auto column_result = ConstOrVectorAdapter<LeftDataType, RightDataType,
+ Operation, is_to_null_type>::
+ execute(block.get_by_position(arguments[0]).column,
+ block.get_by_position(arguments[1]).column, left, right);
+ block.replace_by_position(result, std::move(column_result));
return true;
}
return false;
@@ -711,10 +838,6 @@ public:
return Status::OK();
}
-
- bool can_be_executed_on_default_arguments() const override {
- return CanBeExecutedOnDefaultArguments;
- }
};
} // namespace doris::vectorized
diff --git a/be/src/vec/functions/function_binary_arithmetic_to_null_type.h b/be/src/vec/functions/function_binary_arithmetic_to_null_type.h
deleted file mode 100644
index 66316116ae..0000000000
--- a/be/src/vec/functions/function_binary_arithmetic_to_null_type.h
+++ /dev/null
@@ -1,247 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include "vec/functions/function_binary_arithmetic.h"
-
-namespace doris::vectorized {
-
-/**
- * Arithmetic operations: /, %
- * intDiv (integer division)
- */
-
-template <template <typename, typename> class Op, typename Name,
- bool CanBeExecutedOnDefaultArguments = true>
-class FunctionBinaryArithmeticToNullType : public IFunction {
- bool check_decimal_overflow = true;
-
- template <typename F>
- static bool cast_type(const IDataType* type, F&& f) {
- return cast_type_to_either<DataTypeUInt8, DataTypeUInt16, DataTypeUInt32, DataTypeUInt64,
- DataTypeInt8, DataTypeInt16, DataTypeInt32, DataTypeInt64,
- DataTypeInt128, DataTypeFloat32, DataTypeFloat64,
- DataTypeDecimal<Decimal32>, DataTypeDecimal<Decimal64>,
- DataTypeDecimal<Decimal128>>(type, std::forward<F>(f));
- }
-
- template <typename F>
- static bool cast_both_types(const IDataType* left, const IDataType* right, F&& f) {
- return cast_type(left, [&](const auto& left_) {
- return cast_type(right, [&](const auto& right_) { return f(left_, right_); });
- });
- }
-
-public:
- static constexpr auto name = Name::name;
- static FunctionPtr create() { return std::make_shared<FunctionBinaryArithmeticToNullType>(); }
-
- FunctionBinaryArithmeticToNullType() {}
- String get_name() const override { return name; }
-
- size_t get_number_of_arguments() const override { return 2; }
-
- DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
- DataTypePtr type_res;
-
- const IDataType* first_type = arguments[0].get();
- const IDataType* secord_type = arguments[1].get();
- if (first_type->is_nullable()) {
- first_type = static_cast<const DataTypeNullable*>(first_type)->get_nested_type().get();
- }
- if (secord_type->is_nullable()) {
- secord_type =
- static_cast<const DataTypeNullable*>(secord_type)->get_nested_type().get();
- }
-
- bool valid =
- cast_both_types(first_type, secord_type, [&](const auto& left, const auto& right) {
- using LeftDataType = std::decay_t<decltype(left)>;
- using RightDataType = std::decay_t<decltype(right)>;
- using ResultDataType =
- typename BinaryOperationTraits<Op, LeftDataType,
- RightDataType>::ResultDataType;
- if constexpr (!std::is_same_v<ResultDataType, InvalidType>) {
- if constexpr (IsDataTypeDecimal<LeftDataType> &&
- IsDataTypeDecimal<RightDataType>) {
- constexpr bool is_multiply = false;
- constexpr bool is_division =
- std::is_same_v<Op<UInt8, UInt8>,
- DivideFloatingImpl<UInt8, UInt8>> ||
- std::is_same_v<Op<UInt8, UInt8>,
- DivideIntegralImpl<UInt8, UInt8>> ||
- std::is_same_v<Op<UInt8, UInt8>,
- DivideIntegralOrZeroImpl<UInt8, UInt8>>;
-
- ResultDataType result_type =
- decimal_result_type(left, right, is_multiply, is_division);
- type_res = std::make_shared<ResultDataType>(result_type.get_precision(),
- result_type.get_scale());
- } else if constexpr (IsDataTypeDecimal<LeftDataType>)
- type_res = std::make_shared<LeftDataType>(left.get_precision(),
- left.get_scale());
- else if constexpr (IsDataTypeDecimal<RightDataType>)
- type_res = std::make_shared<RightDataType>(right.get_precision(),
- right.get_scale());
- else if constexpr (IsDataTypeDecimal<ResultDataType>)
- type_res = std::make_shared<ResultDataType>(27, 9);
- else
- type_res = std::make_shared<ResultDataType>();
- return true;
- }
- return false;
- });
- if (!valid) {
- LOG(FATAL) << fmt::format("Illegal types {} and {} of arguments of function {}",
- arguments[0]->get_name(), arguments[1]->get_name(),
- get_name());
- }
-
- return make_nullable(type_res);
- }
-
- bool use_default_implementation_for_nulls() const override { return true; }
-
- Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
- size_t result, size_t input_rows_count) override {
- auto* left_generic = block.get_by_position(arguments[0]).type.get();
- auto* right_generic = block.get_by_position(arguments[1]).type.get();
- if (left_generic->is_nullable()) {
- left_generic =
- static_cast<const DataTypeNullable*>(left_generic)->get_nested_type().get();
- }
- if (right_generic->is_nullable()) {
- right_generic =
- static_cast<const DataTypeNullable*>(right_generic)->get_nested_type().get();
- }
-
- bool valid = cast_both_types(
- left_generic, right_generic, [&](const auto& left, const auto& right) {
- using LeftDataType = std::decay_t<decltype(left)>;
- using RightDataType = std::decay_t<decltype(right)>;
- using ResultDataType =
- typename BinaryOperationTraits<Op, LeftDataType,
- RightDataType>::ResultDataType;
-
- if constexpr (!std::is_same_v<ResultDataType, InvalidType>) {
- constexpr bool result_is_decimal =
- IsDataTypeDecimal<LeftDataType> || IsDataTypeDecimal<RightDataType>;
- constexpr bool is_multiply = false;
- constexpr bool is_division =
- std::is_same_v<Op<UInt8, UInt8>,
- DivideFloatingImpl<UInt8, UInt8>> ||
- std::is_same_v<Op<UInt8, UInt8>,
- DivideIntegralImpl<UInt8, UInt8>> ||
- std::is_same_v<Op<UInt8, UInt8>,
- DivideIntegralOrZeroImpl<UInt8, UInt8>>;
-
- using T0 = typename LeftDataType::FieldType;
- using T1 = typename RightDataType::FieldType;
- using ResultType = typename ResultDataType::FieldType;
- using ColVecT0 = std::conditional_t<IsDecimalNumber<T0>, ColumnDecimal<T0>,
- ColumnVector<T0>>;
- using ColVecT1 = std::conditional_t<IsDecimalNumber<T1>, ColumnDecimal<T1>,
- ColumnVector<T1>>;
- using ColVecResult = std::conditional_t<IsDecimalNumber<ResultType>,
- ColumnDecimal<ResultType>,
- ColumnVector<ResultType>>;
-
- /// Decimal operations need scale. Operations are on result type.
- using OpImpl = std::conditional_t<
- IsDataTypeDecimal<ResultDataType>,
- DecimalBinaryOperation<T0, T1, Op, ResultType>,
- BinaryOperationImpl<T0, T1, Op<T0, T1>, ResultType>>;
-
- auto null_map = ColumnUInt8::create(input_rows_count, 0);
- auto& null_map_data = null_map->get_data();
- size_t argument_size = arguments.size();
- ColumnPtr argument_columns[argument_size];
-
- for (size_t i = 0; i < argument_size; ++i) {
- argument_columns[i] =
- block.get_by_position(arguments[i])
- .column->convert_to_full_column_if_const();
- }
-
- auto col_left_raw = argument_columns[0].get();
- auto col_right_raw = argument_columns[1].get();
-
- typename ColVecResult::MutablePtr col_res = nullptr;
- if constexpr (result_is_decimal) {
- ResultDataType type =
- decimal_result_type(left, right, is_multiply, is_division);
- col_res = ColVecResult::create(0, type.get_scale());
- } else {
- col_res = ColVecResult::create();
- }
-
- auto& vec_res = col_res->get_data();
- vec_res.resize(block.rows());
-
- if (auto col_left = check_and_get_column<ColVecT0>(col_left_raw)) {
- if constexpr (result_is_decimal) {
- ResultDataType type =
- decimal_result_type(left, right, is_multiply, is_division);
-
- typename ResultDataType::FieldType scale_a =
- type.scale_factor_for(left, is_multiply);
- typename ResultDataType::FieldType scale_b =
- type.scale_factor_for(right, is_multiply || is_division);
- if constexpr (IsDataTypeDecimal<RightDataType> && is_division)
- scale_a = right.get_scale_multiplier();
- if (auto col_right =
- check_and_get_column<ColVecT1>(col_right_raw)) {
- OpImpl::vector_vector(col_left->get_data(),
- col_right->get_data(), vec_res, scale_a,
- scale_b, check_decimal_overflow,
- null_map_data);
- }
- } else {
- if (auto col_right =
- check_and_get_column<ColVecT1>(col_right_raw)) {
- OpImpl::vector_vector(col_left->get_data(),
- col_right->get_data(), vec_res,
- null_map_data);
- }
- }
- } else {
- return false;
- }
-
- block.get_by_position(result).column =
- ColumnNullable::create(std::move(col_res), std::move(null_map));
- return true;
- } else {
- return false;
- }
- });
-
- if (!valid) {
- return Status::RuntimeError(
- fmt::format("{}'s arguments do not match the expected data types", get_name()));
- }
-
- return Status::OK();
- }
-
- bool can_be_executed_on_default_arguments() const override {
- return CanBeExecutedOnDefaultArguments;
- }
-};
-
-} // namespace doris::vectorized
diff --git a/be/src/vec/functions/function_bit.cpp b/be/src/vec/functions/function_bit.cpp
index 0f4fb87d27..7af28601a0 100644
--- a/be/src/vec/functions/function_bit.cpp
+++ b/be/src/vec/functions/function_bit.cpp
@@ -20,9 +20,9 @@
#include "vec/data_types/number_traits.h"
#include "vec/functions/function_binary_arithmetic.h"
+#include "vec/functions/function_totype.h"
#include "vec/functions/function_unary_arithmetic.h"
#include "vec/functions/simple_function_factory.h"
-#include "vec/functions/function_totype.h"
namespace doris::vectorized {
@@ -101,10 +101,10 @@ struct BitLengthImpl {
}
};
-using FunctionBitAnd = FunctionBinaryArithmetic<BitAndImpl, NameBitAnd>;
+using FunctionBitAnd = FunctionBinaryArithmetic<BitAndImpl, NameBitAnd, false>;
using FunctionBitNot = FunctionUnaryArithmetic<BitNotImpl, NameBitNot, false>;
-using FunctionBitOr = FunctionBinaryArithmetic<BitOrImpl, NameBitOr>;
-using FunctionBitXor = FunctionBinaryArithmetic<BitXorImpl, NameBitXor>;
+using FunctionBitOr = FunctionBinaryArithmetic<BitOrImpl, NameBitOr, false>;
+using FunctionBitXor = FunctionBinaryArithmetic<BitXorImpl, NameBitXor, false>;
using FunctionBitLength = FunctionUnaryToType<BitLengthImpl, NameBitLength>;
void register_function_bit(SimpleFunctionFactory& factory) {
diff --git a/be/src/vec/functions/function_cast.h b/be/src/vec/functions/function_cast.h
index 144b782554..092842f5fa 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -518,7 +518,6 @@ public:
bool use_default_implementation_for_constants() const override { return true; }
ColumnNumbers get_arguments_that_are_always_constant() const override { return {1}; }
- bool can_be_executed_on_default_arguments() const override { return false; }
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) override {
diff --git a/be/src/vec/functions/int_div.cpp b/be/src/vec/functions/int_div.cpp
index bbb4fd460f..59b2d1c2dc 100644
--- a/be/src/vec/functions/int_div.cpp
+++ b/be/src/vec/functions/int_div.cpp
@@ -18,133 +18,17 @@
// https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/IntDiv.cpp
// and modified by Doris
-#ifdef __SSE2__
-#define LIBDIVIDE_SSE2 1
-#endif
-
#include "vec/functions/int_div.h"
-#include <libdivide.h>
-
#include "vec/functions/function_binary_arithmetic.h"
-#include "vec/functions/function_binary_arithmetic_to_null_type.h"
#include "vec/functions/simple_function_factory.h"
namespace doris::vectorized {
-/// Optimizations for integer division by a constant.
-
-template <typename A, typename B>
-struct DivideIntegralByConstantImpl : BinaryOperationImplBase<A, B, DivideIntegralImpl<A, B>> {
- using ResultType = typename DivideIntegralImpl<A, B>::ResultType;
-
- static void vector_constant(const PaddedPODArray<A>& a, B b, PaddedPODArray<ResultType>& c) {
- // TODO: Support return null in the furture
- if (UNLIKELY(b == 0)) {
-// throw Exception("Division by zero", TStatusCode::VEC_ILLEGAL_DIVISION);
- memset(c.data(), 0, sizeof(ResultType) * c.size());
- return;
- }
-
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wsign-compare"
-
- if (UNLIKELY(std::is_signed_v<B> && b == -1)) {
- size_t size = a.size();
- for (size_t i = 0; i < size; ++i) c[i] = -c[i];
- return;
- }
-
-#pragma GCC diagnostic pop
-
- libdivide::divider<A> divider(b);
-
- size_t size = a.size();
- const A* a_pos = a.data();
- const A* a_end = a_pos + size;
- ResultType* c_pos = c.data();
-
-#ifdef __SSE2__
- static constexpr size_t values_per_sse_register = 16 / sizeof(A);
- const A* a_end_sse = a_pos + size / values_per_sse_register * values_per_sse_register;
-
- while (a_pos < a_end_sse) {
- _mm_storeu_si128(reinterpret_cast<__m128i*>(c_pos),
- _mm_loadu_si128(reinterpret_cast<const __m128i*>(a_pos)) / divider);
-
- a_pos += values_per_sse_register;
- c_pos += values_per_sse_register;
- }
-#endif
-
- while (a_pos < a_end) {
- *c_pos = *a_pos / divider;
- ++a_pos;
- ++c_pos;
- }
- }
-};
-
-/** Specializations are specified for dividing numbers of the type UInt64 and UInt32 by the numbers of the same sign.
- * Can be expanded to all possible combinations, but more code is needed.
- */
-
-template <>
-struct BinaryOperationImpl<UInt64, UInt8, DivideIntegralImpl<UInt64, UInt8>>
- : DivideIntegralByConstantImpl<UInt64, UInt8> {};
-template <>
-struct BinaryOperationImpl<UInt64, UInt16, DivideIntegralImpl<UInt64, UInt16>>
- : DivideIntegralByConstantImpl<UInt64, UInt16> {};
-template <>
-struct BinaryOperationImpl<UInt64, UInt32, DivideIntegralImpl<UInt64, UInt32>>
- : DivideIntegralByConstantImpl<UInt64, UInt32> {};
-template <>
-struct BinaryOperationImpl<UInt64, UInt64, DivideIntegralImpl<UInt64, UInt64>>
- : DivideIntegralByConstantImpl<UInt64, UInt64> {};
-
-template <>
-struct BinaryOperationImpl<UInt32, UInt8, DivideIntegralImpl<UInt32, UInt8>>
- : DivideIntegralByConstantImpl<UInt32, UInt8> {};
-template <>
-struct BinaryOperationImpl<UInt32, UInt16, DivideIntegralImpl<UInt32, UInt16>>
- : DivideIntegralByConstantImpl<UInt32, UInt16> {};
-template <>
-struct BinaryOperationImpl<UInt32, UInt32, DivideIntegralImpl<UInt32, UInt32>>
- : DivideIntegralByConstantImpl<UInt32, UInt32> {};
-template <>
-struct BinaryOperationImpl<UInt32, UInt64, DivideIntegralImpl<UInt32, UInt64>>
- : DivideIntegralByConstantImpl<UInt32, UInt64> {};
-
-template <>
-struct BinaryOperationImpl<Int64, Int8, DivideIntegralImpl<Int64, Int8>>
- : DivideIntegralByConstantImpl<Int64, Int8> {};
-template <>
-struct BinaryOperationImpl<Int64, Int16, DivideIntegralImpl<Int64, Int16>>
- : DivideIntegralByConstantImpl<Int64, Int16> {};
-template <>
-struct BinaryOperationImpl<Int64, Int32, DivideIntegralImpl<Int64, Int32>>
- : DivideIntegralByConstantImpl<Int64, Int32> {};
-template <>
-struct BinaryOperationImpl<Int64, Int64, DivideIntegralImpl<Int64, Int64>>
- : DivideIntegralByConstantImpl<Int64, Int64> {};
-
-template <>
-struct BinaryOperationImpl<Int32, Int8, DivideIntegralImpl<Int32, Int8>>
- : DivideIntegralByConstantImpl<Int32, Int8> {};
-template <>
-struct BinaryOperationImpl<Int32, Int16, DivideIntegralImpl<Int32, Int16>>
- : DivideIntegralByConstantImpl<Int32, Int16> {};
-template <>
-struct BinaryOperationImpl<Int32, Int32, DivideIntegralImpl<Int32, Int32>>
- : DivideIntegralByConstantImpl<Int32, Int32> {};
-template <>
-struct BinaryOperationImpl<Int32, Int64, DivideIntegralImpl<Int32, Int64>>
- : DivideIntegralByConstantImpl<Int32, Int64> {};
-
struct NameIntDiv {
static constexpr auto name = "int_divide";
};
-using FunctionIntDiv = FunctionBinaryArithmeticToNullType<DivideIntegralImpl, NameIntDiv, false>;
+using FunctionIntDiv = FunctionBinaryArithmetic<DivideIntegralImpl, NameIntDiv, true>;
void register_function_int_div(SimpleFunctionFactory& factory) {
factory.register_function<FunctionIntDiv>();
diff --git a/be/src/vec/functions/int_div.h b/be/src/vec/functions/int_div.h
index a74ecab413..7cd5a7d9e3 100644
--- a/be/src/vec/functions/int_div.h
+++ b/be/src/vec/functions/int_div.h
@@ -20,14 +20,15 @@
#pragma once
-#include "common/compiler_util.h"
-#include "common/logging.h"
-#include "common/status.h"
-#include "runtime/decimalv2_value.h"
-#include "type_traits"
+#include <libdivide.h>
+
+#include <type_traits>
+
+#include "vec/columns/column_decimal.h"
#include "vec/columns/column_nullable.h"
-#include "vec/common/exception.h"
+#include "vec/core/types.h"
#include "vec/data_types/number_traits.h"
+#include "vec/functions/function_binary_arithmetic.h"
namespace doris::vectorized {
@@ -38,18 +39,34 @@ namespace doris::vectorized {
template <typename A, typename B>
struct DivideIntegralImpl {
using ResultType = typename NumberTraits::ResultOfIntegerDivision<A, B>::Type;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
+
+ template <typename Result = ResultType>
+ static void apply(const typename Traits::ArrayA& a, B b,
+ typename ColumnVector<Result>::Container& c,
+ typename Traits::ArrayNull& null_map) {
+ size_t size = c.size();
+ UInt8 is_null = b == 0;
+ memset(null_map.data(), is_null, size);
+
+ if (!is_null) {
+ if constexpr (!std::is_floating_point_v<A> && !std::is_same_v<A, Int128> &&
+ !std::is_same_v<A, Int8> && !std::is_same_v<A, UInt8>) {
+ for (size_t i = 0; i < size; i++) {
+ c[i] = a[i] / libdivide::divider<A>(b);
+ }
+ } else {
+ for (size_t i = 0; i < size; i++) {
+ c[i] = a[i] / b;
+ }
+ }
+ }
+ }
template <typename Result = ResultType>
- static inline Result apply(A a, B b, NullMap& null_map, size_t index) {
- null_map[index] = b == 0;
-
- /// Otherwise overflow may occur due to integer promotion. Example: int8_t(-1) / uint64_t(2).
- /// NOTE: overflow is still possible when dividing large signed number to large unsigned number or vice-versa. But it's less harmful.
- if constexpr (std::is_integral_v<A> && std::is_integral_v<B> &&
- (std::is_signed_v<A> || std::is_signed_v<B>))
- return std::make_signed_t<A>(a) / (std::make_signed_t<B>(b) + (b == 0));
- else
- return a / (b + (b == 0));
+ static inline Result apply(A a, B b, UInt8& is_null) {
+ is_null = b == 0;
+ return a / (b + is_null);
}
};
diff --git a/be/src/vec/functions/math.cpp b/be/src/vec/functions/math.cpp
index 92ef20fe93..03bdeaf307 100644
--- a/be/src/vec/functions/math.cpp
+++ b/be/src/vec/functions/math.cpp
@@ -15,10 +15,11 @@
// specific language governing permissions and limitations
// under the License.
+#include "vec/core/types.h"
#include "vec/data_types/number_traits.h"
#include "vec/functions/function_const.h"
#include "vec/functions/function_binary_arithmetic.h"
-#include "vec/functions/function_binary_arithmetic_to_null_type.h"
+#include "vec/functions/function_binary_arithmetic.h"
#include "vec/functions/function_math_unary.h"
#include "vec/functions/function_math_unary_to_null_type.h"
#include "vec/functions/function_string.h"
@@ -150,17 +151,39 @@ struct LogName {
template <typename A, typename B>
struct LogImpl {
using ResultType = Float64;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
+
static const constexpr bool allow_decimal = false;
+ static constexpr double EPSILON = 1e-9;
+
+ template <typename Result = ResultType>
+ static void apply(const typename Traits::ArrayA& a, B b,
+ typename ColumnVector<Result>::Container& c,
+ typename Traits::ArrayNull& null_map) {
+ size_t size = c.size();
+ UInt8 is_null = b <= 0;
+ memset(null_map.data(), is_null, size);
+
+ if (!is_null) {
+ for (size_t i = 0; i < size; i++) {
+ if (a[i] <= 0 || std::fabs(a[i] - 1.0) < EPSILON) {
+ null_map[i] = 1;
+ } else {
+ c[i] = static_cast<Float64>(std::log(static_cast<Float64>(b)) /
+ std::log(static_cast<Float64>(a[i])));
+ }
+ }
+ }
+ }
template <typename Result>
- static inline Result apply(A a, B b, NullMap& null_map, size_t index) {
- constexpr double EPSILON = 1e-9;
- null_map[index] = a <= 0 || b <= 0 || std::fabs(a - 1.0) < EPSILON;
+ static inline Result apply(A a, B b, UInt8& is_null) {
+ is_null = a <= 0 || b <= 0 || std::fabs(a - 1.0) < EPSILON;
return static_cast<Float64>(std::log(static_cast<Float64>(b)) /
std::log(static_cast<Float64>(a)));
}
};
-using FunctionLog = FunctionBinaryArithmeticToNullType<LogImpl, LogName>;
+using FunctionLog = FunctionBinaryArithmetic<LogImpl, LogName, true>;
struct CeilName {
static constexpr auto name = "ceil";
@@ -359,7 +382,7 @@ struct PowImpl {
struct PowName {
static constexpr auto name = "pow";
};
-using FunctionPow = FunctionBinaryArithmetic<PowImpl, PowName>;
+using FunctionPow = FunctionBinaryArithmetic<PowImpl, PowName, false>;
template <typename A, typename B>
struct TruncateImpl {
@@ -376,7 +399,7 @@ struct TruncateImpl {
struct TruncateName {
static constexpr auto name = "truncate";
};
-using FunctionTruncate = FunctionBinaryArithmetic<TruncateImpl, TruncateName>;
+using FunctionTruncate = FunctionBinaryArithmetic<TruncateImpl, TruncateName, false>;
/// round(double,int32)-->double
/// key_str:roundFloat64Int32
@@ -397,7 +420,7 @@ struct RoundTwoImpl {
static_cast<Float64>(a), static_cast<Int32>(b), false, false));
}
};
-using FunctionRoundTwo = FunctionBinaryArithmetic<RoundTwoImpl, RoundName>;
+using FunctionRoundTwo = FunctionBinaryArithmetic<RoundTwoImpl, RoundName, false>;
// TODO: Now math may cause one thread compile time too long, because the function in math
// so mush. Split it to speed up compile time in the future
diff --git a/be/src/vec/functions/minus.cpp b/be/src/vec/functions/minus.cpp
index 52b86a1085..e282935729 100644
--- a/be/src/vec/functions/minus.cpp
+++ b/be/src/vec/functions/minus.cpp
@@ -49,7 +49,7 @@ struct MinusImpl {
struct NameMinus {
static constexpr auto name = "subtract";
};
-using FunctionMinus = FunctionBinaryArithmetic<MinusImpl, NameMinus>;
+using FunctionMinus = FunctionBinaryArithmetic<MinusImpl, NameMinus, false>;
void register_function_minus(SimpleFunctionFactory& factory) {
factory.register_function<FunctionMinus>();
diff --git a/be/src/vec/functions/modulo.cpp b/be/src/vec/functions/modulo.cpp
index 28b0ec768d..599790d0d4 100644
--- a/be/src/vec/functions/modulo.cpp
+++ b/be/src/vec/functions/modulo.cpp
@@ -18,16 +18,12 @@
// https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/Modulo.cpp
// and modified by Doris
-#include "runtime/decimalv2_value.h"
-#ifdef __SSE2__
-#define LIBDIVIDE_SSE2 1
-#endif
-
#include <libdivide.h>
#include "common/status.h"
+#include "runtime/decimalv2_value.h"
+#include "vec/core/types.h"
#include "vec/functions/function_binary_arithmetic.h"
-#include "vec/functions/function_binary_arithmetic_to_null_type.h"
#include "vec/functions/simple_function_factory.h"
namespace doris::vectorized {
@@ -35,144 +31,90 @@ namespace doris::vectorized {
template <typename A, typename B>
struct ModuloImpl {
using ResultType = typename NumberTraits::ResultOfModulo<A, B>::Type;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
+
+ template <typename Result = ResultType>
+ static void apply(const typename Traits::ArrayA& a, B b,
+ typename ColumnVector<Result>::Container& c,
+ typename Traits::ArrayNull& null_map) {
+ size_t size = c.size();
+ UInt8 is_null = b == 0;
+ memset(null_map.data(), is_null, sizeof(UInt8) * size);
+
+ if (!is_null) {
+ for (size_t i = 0; i < size; i++) {
+ if constexpr (std::is_floating_point_v<ResultType>) {
+ c[i] = std::fmod((double)a[i], (double)b);
+ } else {
+ c[i] = a[i] % b;
+ }
+ }
+ }
+ }
template <typename Result = ResultType>
- static inline Result apply(A a, B b, NullMap& null_map, size_t index) {
+ static inline Result apply(A a, B b, UInt8& is_null) {
+ is_null = b == 0;
+ b += is_null;
+
if constexpr (std::is_floating_point_v<Result>) {
- null_map[index] = b == 0;
return std::fmod((double)a, (double)b);
} else {
- null_map[index] = b == 0;
- return typename NumberTraits::ToInteger<A>::Type(a) %
- (typename NumberTraits::ToInteger<B>::Type(b) + (b == 0));
+ return a % b;
}
}
template <typename Result = DecimalV2Value>
- static inline DecimalV2Value apply(DecimalV2Value a, DecimalV2Value b, NullMap& null_map,
- size_t index) {
- null_map[index] = b == DecimalV2Value(0);
- return a % (b + DecimalV2Value(b == DecimalV2Value(0)));
+ static inline DecimalV2Value apply(DecimalV2Value a, DecimalV2Value b, UInt8& is_null) {
+ is_null = b == DecimalV2Value(0);
+ return a % (b + DecimalV2Value(is_null));
}
};
template <typename A, typename B>
struct PModuloImpl {
using ResultType = typename NumberTraits::ResultOfModulo<A, B>::Type;
+ using Traits = NumberTraits::BinaryOperatorTraits<A, B>;
+
+ template <typename Result = ResultType>
+ static void apply(const typename Traits::ArrayA& a, B b,
+ typename ColumnVector<Result>::Container& c,
+ typename Traits::ArrayNull& null_map) {
+ size_t size = c.size();
+ UInt8 is_null = b == 0;
+ memset(null_map.data(), is_null, size);
+
+ if (!is_null) {
+ for (size_t i = 0; i < size; i++) {
+ if constexpr (std::is_floating_point_v<ResultType>) {
+ c[i] = std::fmod(std::fmod((double)a[i], (double)b) + (double)b, double(b));
+ } else {
+ c[i] = (a[i] % b + b) % b;
+ }
+ }
+ }
+ }
template <typename Result = ResultType>
- static inline Result apply(A a, B b, NullMap& null_map, size_t index) {
+ static inline Result apply(A a, B b, UInt8& is_null) {
+ is_null = b == 0;
+ b += is_null;
+
if constexpr (std::is_floating_point_v<Result>) {
- null_map[index] = 0;
return std::fmod(std::fmod((double)a, (double)b) + (double)b, (double)b);
} else {
- null_map[index] = b == 0;
- return (typename NumberTraits::ToInteger<A>::Type(a) %
- (typename NumberTraits::ToInteger<B>::Type(b) + (b == 0)) +
- typename NumberTraits::ToInteger<B>::Type(b)) %
- (typename NumberTraits::ToInteger<B>::Type(b) + (b == 0));
+ return (a % b + b) % b;
}
}
template <typename Result = DecimalV2Value>
- static inline DecimalV2Value apply(DecimalV2Value a, DecimalV2Value b, NullMap& null_map,
- size_t index) {
- null_map[index] = b == DecimalV2Value(0);
- return (a % (b + DecimalV2Value(b == DecimalV2Value(0))) + b) %
- (b + DecimalV2Value(b == DecimalV2Value(0)));
+ static inline DecimalV2Value apply(DecimalV2Value a, DecimalV2Value b, UInt8& is_null) {
+ is_null = b == DecimalV2Value(0);
+ b += DecimalV2Value(is_null);
+ return (a % b + b) % b;
}
};
-template <typename A, typename B>
-struct ModuloByConstantImpl : BinaryOperationImplBase<A, B, ModuloImpl<A, B>> {
- using ResultType = typename ModuloImpl<A, B>::ResultType;
-
- static void vector_constant(const PaddedPODArray<A>& a, B b, PaddedPODArray<ResultType>& c) {
- // TODO: Support return NULL in the future
- if (UNLIKELY(b == 0)) {
- // throw Exception("Division by zero", TStatusCode::VEC_ILLEGAL_DIVISION);
- memset(c.data(), 0, sizeof(ResultType) * c.size());
- return;
- }
-
-#pragma GCC diagnostic push
-#pragma GCC diagnostic ignored "-Wsign-compare"
-
- if (UNLIKELY((std::is_signed_v<B> && b == -1) || b == 1)) {
- size_t size = a.size();
- for (size_t i = 0; i < size; ++i) c[i] = 0;
- return;
- }
-
-#pragma GCC diagnostic pop
-
- libdivide::divider<A> divider(b);
-
- /// Here we failed to make the SSE variant from libdivide give an advantage.
- size_t size = a.size();
- for (size_t i = 0; i < size; ++i)
- c[i] = a[i] -
- (a[i] / divider) *
- b; /// NOTE: perhaps, the division semantics with the remainder of negative numbers is not preserved.
- }
-};
-
-/** Specializations are specified for dividing numbers of the type UInt64 and UInt32 by the numbers of the same sign.
- * Can be expanded to all possible combinations, but more code is needed.
- */
-
-template <>
-struct BinaryOperationImpl<UInt64, UInt8, ModuloImpl<UInt64, UInt8>>
- : ModuloByConstantImpl<UInt64, UInt8> {};
-template <>
-struct BinaryOperationImpl<UInt64, UInt16, ModuloImpl<UInt64, UInt16>>
- : ModuloByConstantImpl<UInt64, UInt16> {};
-template <>
-struct BinaryOperationImpl<UInt64, UInt32, ModuloImpl<UInt64, UInt32>>
- : ModuloByConstantImpl<UInt64, UInt32> {};
-template <>
-struct BinaryOperationImpl<UInt64, UInt64, ModuloImpl<UInt64, UInt64>>
- : ModuloByConstantImpl<UInt64, UInt64> {};
-
-template <>
-struct BinaryOperationImpl<UInt32, UInt8, ModuloImpl<UInt32, UInt8>>
- : ModuloByConstantImpl<UInt32, UInt8> {};
-template <>
-struct BinaryOperationImpl<UInt32, UInt16, ModuloImpl<UInt32, UInt16>>
- : ModuloByConstantImpl<UInt32, UInt16> {};
-template <>
-struct BinaryOperationImpl<UInt32, UInt32, ModuloImpl<UInt32, UInt32>>
- : ModuloByConstantImpl<UInt32, UInt32> {};
-template <>
-struct BinaryOperationImpl<UInt32, UInt64, ModuloImpl<UInt32, UInt64>>
- : ModuloByConstantImpl<UInt32, UInt64> {};
-
-template <>
-struct BinaryOperationImpl<Int64, Int8, ModuloImpl<Int64, Int8>>
- : ModuloByConstantImpl<Int64, Int8> {};
-template <>
-struct BinaryOperationImpl<Int64, Int16, ModuloImpl<Int64, Int16>>
- : ModuloByConstantImpl<Int64, Int16> {};
-template <>
-struct BinaryOperationImpl<Int64, Int32, ModuloImpl<Int64, Int32>>
- : ModuloByConstantImpl<Int64, Int32> {};
-template <>
-struct BinaryOperationImpl<Int64, Int64, ModuloImpl<Int64, Int64>>
- : ModuloByConstantImpl<Int64, Int64> {};
-
-template <>
-struct BinaryOperationImpl<Int32, Int8, ModuloImpl<Int32, Int8>>
- : ModuloByConstantImpl<Int32, Int8> {};
-template <>
-struct BinaryOperationImpl<Int32, Int16, ModuloImpl<Int32, Int16>>
- : ModuloByConstantImpl<Int32, Int16> {};
-template <>
-struct BinaryOperationImpl<Int32, Int32, ModuloImpl<Int32, Int32>>
- : ModuloByConstantImpl<Int32, Int32> {};
-template <>
-struct BinaryOperationImpl<Int32, Int64, ModuloImpl<Int32, Int64>>
- : ModuloByConstantImpl<Int32, Int64> {};
-
struct NameModulo {
static constexpr auto name = "mod";
};
@@ -180,8 +122,8 @@ struct NamePModulo {
static constexpr auto name = "pmod";
};
-using FunctionModulo = FunctionBinaryArithmeticToNullType<ModuloImpl, NameModulo, false>;
-using FunctionPModulo = FunctionBinaryArithmeticToNullType<PModuloImpl, NamePModulo, false>;
+using FunctionModulo = FunctionBinaryArithmetic<ModuloImpl, NameModulo, true>;
+using FunctionPModulo = FunctionBinaryArithmetic<PModuloImpl, NamePModulo, true>;
void register_function_modulo(SimpleFunctionFactory& factory) {
factory.register_function<FunctionModulo>();
diff --git a/be/src/vec/functions/multiply.cpp b/be/src/vec/functions/multiply.cpp
index b2840e42e5..95475ed847 100644
--- a/be/src/vec/functions/multiply.cpp
+++ b/be/src/vec/functions/multiply.cpp
@@ -48,7 +48,7 @@ struct MultiplyImpl {
struct NameMultiply {
static constexpr auto name = "multiply";
};
-using FunctionMultiply = FunctionBinaryArithmetic<MultiplyImpl, NameMultiply>;
+using FunctionMultiply = FunctionBinaryArithmetic<MultiplyImpl, NameMultiply, false>;
void register_function_multiply(SimpleFunctionFactory& factory) {
factory.register_function<FunctionMultiply>();
diff --git a/be/src/vec/functions/plus.cpp b/be/src/vec/functions/plus.cpp
index 0da30dfa20..d573d3a3d6 100644
--- a/be/src/vec/functions/plus.cpp
+++ b/be/src/vec/functions/plus.cpp
@@ -50,7 +50,7 @@ struct PlusImpl {
struct NamePlus {
static constexpr auto name = "add";
};
-using FunctionPlus = FunctionBinaryArithmetic<PlusImpl, NamePlus>;
+using FunctionPlus = FunctionBinaryArithmetic<PlusImpl, NamePlus, false>;
void register_function_plus(SimpleFunctionFactory& factory) {
factory.register_function<FunctionPlus>();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 05/05: [Bug] (fix) DeltaWriter::mem_consumption() coredump (#9245)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 162b8492e1acc6413deb9ae4ed4095615b13f8e1
Author: chenlinzhong <49...@qq.com>
AuthorDate: Sat May 7 19:13:08 2022 +0800
[Bug] (fix) DeltaWriter::mem_consumption() coredump (#9245)
---
be/src/olap/delta_writer.cpp | 9 +++++++++
be/src/olap/delta_writer.h | 7 +++++++
be/src/runtime/tablets_channel.cpp | 8 ++++----
3 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index 9738f8f55b..8446340d5a 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -338,6 +338,15 @@ OLAPStatus DeltaWriter::cancel() {
return OLAP_SUCCESS;
}
+int64_t DeltaWriter::save_mem_consumption_snapshot() {
+ _mem_consumption_snapshot = mem_consumption();
+ return _mem_consumption_snapshot;
+}
+
+int64_t DeltaWriter::get_mem_consumption_snapshot() const {
+ return _mem_consumption_snapshot;
+}
+
int64_t DeltaWriter::mem_consumption() const {
if (_mem_tracker == nullptr) {
// This method may be called before this writer is initialized.
diff --git a/be/src/olap/delta_writer.h b/be/src/olap/delta_writer.h
index dbea5694a7..c765d03115 100644
--- a/be/src/olap/delta_writer.h
+++ b/be/src/olap/delta_writer.h
@@ -89,6 +89,10 @@ public:
int64_t tablet_id() { return _tablet->tablet_id(); }
+ int64_t save_mem_consumption_snapshot();
+
+ int64_t get_mem_consumption_snapshot() const;
+
private:
DeltaWriter(WriteRequest* req, const std::shared_ptr<MemTracker>& parent,
StorageEngine* storage_engine);
@@ -121,6 +125,9 @@ private:
int64_t _segment_counter = 0;
std::mutex _lock;
+
+ //only used for std::sort more detail see issue(#9237)
+ int64_t _mem_consumption_snapshot = 0;
};
} // namespace doris
diff --git a/be/src/runtime/tablets_channel.cpp b/be/src/runtime/tablets_channel.cpp
index e8e2990267..75f9936e9b 100644
--- a/be/src/runtime/tablets_channel.cpp
+++ b/be/src/runtime/tablets_channel.cpp
@@ -216,12 +216,12 @@ Status TabletsChannel::reduce_mem_usage(int64_t mem_limit) {
// Sort the DeltaWriters by mem consumption in descend order.
std::vector<DeltaWriter*> writers;
for (auto& it : _tablet_writers) {
+ it.second->save_mem_consumption_snapshot();
writers.push_back(it.second);
}
- std::sort(writers.begin(), writers.end(),
- [](const DeltaWriter* lhs, const DeltaWriter* rhs) {
- return lhs->mem_consumption() > rhs->mem_consumption();
- });
+ std::sort(writers.begin(), writers.end(), [](const DeltaWriter* lhs, const DeltaWriter* rhs) {
+ return lhs->get_mem_consumption_snapshot() > rhs->get_mem_consumption_snapshot();
+ });
// Decide which writes should be flushed to reduce mem consumption.
// The main idea is to flush at least one third of the mem_limit.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 04/05: [enhancement](load) optimize load string data and dict page write (#9123)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 77932f066c1d857a0cc251547ab53c1a572c77ca
Author: Zhengguo Yang <ya...@gmail.com>
AuthorDate: Sat May 7 10:27:27 2022 +0800
[enhancement](load) optimize load string data and dict page write (#9123)
* [enhancement](load) optimize load string data and dict page write
---
be/src/olap/memtable.cpp | 83 ++++++++++++----------
be/src/olap/memtable.h | 10 ++-
be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 34 ++++-----
be/src/olap/rowset/segment_v2/binary_plain_page.h | 18 ++---
be/src/olap/rowset/segment_v2/bitshuffle_page.h | 37 +++++++++-
5 files changed, 114 insertions(+), 68 deletions(-)
diff --git a/be/src/olap/memtable.cpp b/be/src/olap/memtable.cpp
index 460638d836..a7d6728ebe 100644
--- a/be/src/olap/memtable.cpp
+++ b/be/src/olap/memtable.cpp
@@ -43,18 +43,27 @@ MemTable::MemTable(int64_t tablet_id, Schema* schema, const TabletSchema* tablet
_table_mem_pool(new MemPool(_mem_tracker.get())),
_schema_size(_schema->schema_size()),
_rowset_writer(rowset_writer) {
+ if (_keys_type == KeysType::DUP_KEYS) {
+ _insert_fn = &MemTable::_insert_dup;
+ } else {
+ _insert_fn = &MemTable::_insert_agg;
+ }
+ if (_tablet_schema->has_sequence_col()) {
+ _aggregate_two_row_fn = &MemTable::_aggregate_two_row_with_sequence;
+ } else {
+ _aggregate_two_row_fn = &MemTable::_aggregate_two_row;
+ }
if (tablet_schema->sort_type() == SortType::ZORDER) {
- _row_comparator =
- std::make_shared<TupleRowZOrderComparator>(_schema, tablet_schema->sort_col_num());
+ _row_comparator = std::make_shared<TupleRowZOrderComparator>(
+ _schema, tablet_schema->sort_col_num());
} else {
_row_comparator = std::make_shared<RowCursorComparator>(_schema);
}
- _skip_list = new Table(_row_comparator.get(), _table_mem_pool.get(),
- _keys_type == KeysType::DUP_KEYS);
+ _skip_list = std::make_unique<Table>(_row_comparator.get(), _table_mem_pool.get(),
+ _keys_type == KeysType::DUP_KEYS);
}
MemTable::~MemTable() {
- delete _skip_list;
}
MemTable::RowCursorComparator::RowCursorComparator(const Schema* schema) : _schema(schema) {}
@@ -65,37 +74,25 @@ int MemTable::RowCursorComparator::operator()(const char* left, const char* righ
return compare_row(lhs_row, rhs_row);
}
-void MemTable::insert(const Tuple* tuple) {
+// For non-DUP models, for the data rows passed from the upper layer, when copying the data,
+// we first allocate from _buffer_mem_pool, and then check whether it already exists in
+// _skiplist. If it exists, we aggregate the new row into the row in skiplist.
+// otherwise, we need to copy it into _table_mem_pool before we can insert it.
+void MemTable::_insert_agg(const Tuple* tuple) {
_rows++;
- bool overwritten = false;
- uint8_t* _tuple_buf = nullptr;
- if (_keys_type == KeysType::DUP_KEYS) {
- // Will insert directly, so use memory from _table_mem_pool
- _tuple_buf = _table_mem_pool->allocate(_schema_size);
- ContiguousRow row(_schema, _tuple_buf);
- _tuple_to_row(tuple, &row, _table_mem_pool.get());
- _skip_list->Insert((TableKey)_tuple_buf, &overwritten);
- DCHECK(!overwritten) << "Duplicate key model meet overwrite in SkipList";
- return;
- }
-
- // For non-DUP models, for the data rows passed from the upper layer, when copying the data,
- // we first allocate from _buffer_mem_pool, and then check whether it already exists in
- // _skiplist. If it exists, we aggregate the new row into the row in skiplist.
- // otherwise, we need to copy it into _table_mem_pool before we can insert it.
- _tuple_buf = _buffer_mem_pool->allocate(_schema_size);
- ContiguousRow src_row(_schema, _tuple_buf);
+ uint8_t* tuple_buf = _buffer_mem_pool->allocate(_schema_size);
+ ContiguousRow src_row(_schema, tuple_buf);
_tuple_to_row(tuple, &src_row, _buffer_mem_pool.get());
- bool is_exist = _skip_list->Find((TableKey)_tuple_buf, &_hint);
+ bool is_exist = _skip_list->Find((TableKey)tuple_buf, &_hint);
if (is_exist) {
- _aggregate_two_row(src_row, _hint.curr->key);
+ (this->*_aggregate_two_row_fn)(src_row, _hint.curr->key);
} else {
- _tuple_buf = _table_mem_pool->allocate(_schema_size);
- ContiguousRow dst_row(_schema, _tuple_buf);
+ tuple_buf = _table_mem_pool->allocate(_schema_size);
+ ContiguousRow dst_row(_schema, tuple_buf);
_agg_object_pool.acquire_data(&_agg_buffer_pool);
copy_row_in_memtable(&dst_row, src_row, _table_mem_pool.get());
- _skip_list->InsertWithHint((TableKey)_tuple_buf, is_exist, &_hint);
+ _skip_list->InsertWithHint((TableKey)tuple_buf, is_exist, &_hint);
}
// Make MemPool to be reusable, but does not free its memory
@@ -103,6 +100,16 @@ void MemTable::insert(const Tuple* tuple) {
_agg_buffer_pool.clear();
}
+void MemTable::_insert_dup(const Tuple* tuple) {
+ _rows++;
+ bool overwritten = false;
+ uint8_t* tuple_buf = _table_mem_pool->allocate(_schema_size);
+ ContiguousRow row(_schema, tuple_buf);
+ _tuple_to_row(tuple, &row, _table_mem_pool.get());
+ _skip_list->Insert((TableKey)tuple_buf, &overwritten);
+ DCHECK(!overwritten) << "Duplicate key model meet overwrite in SkipList";
+}
+
void MemTable::_tuple_to_row(const Tuple* tuple, ContiguousRow* row, MemPool* mem_pool) {
for (size_t i = 0; i < _slot_descs->size(); ++i) {
auto cell = row->cell(i);
@@ -116,12 +123,14 @@ void MemTable::_tuple_to_row(const Tuple* tuple, ContiguousRow* row, MemPool* me
void MemTable::_aggregate_two_row(const ContiguousRow& src_row, TableKey row_in_skiplist) {
ContiguousRow dst_row(_schema, row_in_skiplist);
- if (_tablet_schema->has_sequence_col()) {
- agg_update_row_with_sequence(&dst_row, src_row, _tablet_schema->sequence_col_idx(),
- _table_mem_pool.get());
- } else {
- agg_update_row(&dst_row, src_row, _table_mem_pool.get());
- }
+ agg_update_row(&dst_row, src_row, _table_mem_pool.get());
+}
+
+void MemTable::_aggregate_two_row_with_sequence(const ContiguousRow& src_row,
+ TableKey row_in_skiplist) {
+ ContiguousRow dst_row(_schema, row_in_skiplist);
+ agg_update_row_with_sequence(&dst_row, src_row, _tablet_schema->sequence_col_idx(),
+ _table_mem_pool.get());
}
OLAPStatus MemTable::flush() {
@@ -134,7 +143,7 @@ OLAPStatus MemTable::flush() {
if (st == OLAP_ERR_FUNC_NOT_IMPLEMENTED) {
// For alpha rowset, we do not implement "flush_single_memtable".
// Flush the memtable like the old way.
- Table::Iterator it(_skip_list);
+ Table::Iterator it(_skip_list.get());
for (it.SeekToFirst(); it.Valid(); it.Next()) {
char* row = (char*)it.key();
ContiguousRow dst_row(_schema, row);
@@ -158,7 +167,7 @@ OLAPStatus MemTable::close() {
}
MemTable::Iterator::Iterator(MemTable* memtable)
- : _mem_table(memtable), _it(memtable->_skip_list) {}
+ : _mem_table(memtable), _it(memtable->_skip_list.get()) {}
void MemTable::Iterator::seek_to_first() {
_it.SeekToFirst();
diff --git a/be/src/olap/memtable.h b/be/src/olap/memtable.h
index 4282ba5c06..d400b74903 100644
--- a/be/src/olap/memtable.h
+++ b/be/src/olap/memtable.h
@@ -46,7 +46,7 @@ public:
int64_t tablet_id() const { return _tablet_id; }
size_t memory_usage() const { return _mem_tracker->consumption(); }
- void insert(const Tuple* tuple);
+ inline void insert(const Tuple* tuple) { (this->*_insert_fn)(tuple); }
/// Flush
OLAPStatus flush();
OLAPStatus close();
@@ -88,6 +88,9 @@ public:
private:
void _tuple_to_row(const Tuple* tuple, ContiguousRow* row, MemPool* mem_pool);
void _aggregate_two_row(const ContiguousRow& new_row, TableKey row_in_skiplist);
+ void _aggregate_two_row_with_sequence(const ContiguousRow& new_row, TableKey row_in_skiplist);
+ void _insert_dup(const Tuple* tuple);
+ void _insert_agg(const Tuple* tuple);
int64_t _tablet_id;
Schema* _schema;
@@ -112,7 +115,7 @@ private:
ObjectPool _agg_object_pool;
size_t _schema_size;
- Table* _skip_list;
+ std::unique_ptr<Table> _skip_list;
Table::Hint _hint;
RowsetWriter* _rowset_writer;
@@ -123,6 +126,9 @@ private:
// This is not the rows in this memtable, because rows may be merged
// in unique or aggragate key model.
int64_t _rows = 0;
+ void (MemTable::*_insert_fn)(const Tuple* tuple) = nullptr;
+ void (MemTable::*_aggregate_two_row_fn)(const ContiguousRow& new_row,
+ TableKey row_in_skiplist) = nullptr;
}; // class MemTable
diff --git a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
index 95a9fd5c7b..9cee35f997 100644
--- a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
+++ b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
@@ -23,9 +23,9 @@
#include "util/slice.h" // for Slice
#include "vec/columns/column.h"
#include "vec/columns/column_dictionary.h"
-#include "vec/columns/column_vector.h"
-#include "vec/columns/column_string.h"
#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
#include "vec/columns/predicate_column.h"
namespace doris {
@@ -67,6 +67,8 @@ Status BinaryDictPageBuilder::add(const uint8_t* vals, size_t* count) {
const Slice* src = reinterpret_cast<const Slice*>(vals);
size_t num_added = 0;
uint32_t value_code = -1;
+ auto* actual_builder =
+ down_cast<BitshufflePageBuilder<OLAP_FIELD_TYPE_INT>*>(_data_page_builder.get());
if (_data_page_builder->count() == 0) {
_first_value.assign_copy(reinterpret_cast<const uint8_t*>(src->get_data()),
@@ -91,13 +93,18 @@ Status BinaryDictPageBuilder::add(const uint8_t* vals, size_t* count) {
dict_item.relocate(item_mem);
}
value_code = _dictionary.size();
+ size_t add_count = 1;
+ RETURN_IF_ERROR(_dict_builder->add(reinterpret_cast<const uint8_t*>(&dict_item),
+ &add_count));
+ if (add_count == 0) {
+ // current dict page is full, stop processing remaining inputs
+ break;
+ }
_dictionary.emplace(dict_item, value_code);
- _dict_items.push_back(dict_item);
- _dict_builder->update_prepared_size(dict_item.size);
}
size_t add_count = 1;
- RETURN_IF_ERROR(_data_page_builder->add(reinterpret_cast<const uint8_t*>(&value_code),
- &add_count));
+ RETURN_IF_ERROR(actual_builder->single_add(
+ reinterpret_cast<const uint8_t*>(&value_code), &add_count));
if (add_count == 0) {
// current data page is full, stop processing remaining inputs
break;
@@ -145,17 +152,7 @@ uint64_t BinaryDictPageBuilder::size() const {
}
Status BinaryDictPageBuilder::get_dictionary_page(OwnedSlice* dictionary_page) {
- _dictionary.clear();
- _dict_builder->reset();
- size_t add_count = 1;
- // here do not check is_page_full of dict_builder
- // because it is checked in add
- for (auto& dict_item : _dict_items) {
- RETURN_IF_ERROR(
- _dict_builder->add(reinterpret_cast<const uint8_t*>(&dict_item), &add_count));
- }
*dictionary_page = _dict_builder->finish();
- _dict_items.clear();
return Status::OK();
}
@@ -181,10 +178,7 @@ Status BinaryDictPageBuilder::get_last_value(void* value) const {
}
uint32_t value_code;
RETURN_IF_ERROR(_data_page_builder->get_last_value(&value_code));
- // TODO _dict_items is cleared in get_dictionary_page, which could cause
- // get_last_value to fail when it's called after get_dictionary_page.
- // the solution is to read last value from _dict_builder instead of _dict_items
- *reinterpret_cast<Slice*>(value) = _dict_items[value_code];
+ *reinterpret_cast<Slice*>(value) = _dict_builder->get(value_code);
return Status::OK();
}
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 061884fb2b..5658d1cb77 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -47,14 +47,13 @@ namespace segment_v2 {
class BinaryPlainPageBuilder : public PageBuilder {
public:
BinaryPlainPageBuilder(const PageBuilderOptions& options)
- : _size_estimate(0), _prepared_size(0), _options(options) {
+ : _size_estimate(0), _options(options) {
reset();
}
bool is_page_full() override {
// data_page_size is 0, do not limit the page size
- return _options.data_page_size != 0 && (_size_estimate > _options.data_page_size ||
- _prepared_size > _options.data_page_size);
+ return _options.data_page_size != 0 && _size_estimate > _options.data_page_size;
}
Status add(const uint8_t* vals, size_t* count) override {
@@ -101,7 +100,6 @@ public:
_buffer.clear();
_buffer.reserve(_options.data_page_size == 0 ? 1024 : _options.data_page_size);
_size_estimate = sizeof(uint32_t);
- _prepared_size = sizeof(uint32_t);
_finished = false;
_last_value_size = 0;
}
@@ -127,11 +125,16 @@ public:
return Status::OK();
}
- void update_prepared_size(size_t added_size) {
- _prepared_size += added_size;
- _prepared_size += sizeof(uint32_t);
+ inline Slice operator[](size_t idx) const {
+ DCHECK(!_finished);
+ DCHECK_LT(idx, _offsets.size());
+ size_t value_size =
+ (idx < _offsets.size() - 1) ? _offsets[idx + 1] - _offsets[idx] : _last_value_size;
+ return Slice(&_buffer[_offsets[idx]], value_size);
}
+ inline Slice get(std::size_t idx) const { return (*this)[idx]; }
+
private:
void _copy_value_at(size_t idx, faststring* value) const {
size_t value_size =
@@ -141,7 +144,6 @@ private:
faststring _buffer;
size_t _size_estimate;
- size_t _prepared_size;
// Offsets of each entry, relative to the start of the page
std::vector<uint32_t> _offsets;
bool _finished;
diff --git a/be/src/olap/rowset/segment_v2/bitshuffle_page.h b/be/src/olap/rowset/segment_v2/bitshuffle_page.h
index fe7ba566fa..9bada3f764 100644
--- a/be/src/olap/rowset/segment_v2/bitshuffle_page.h
+++ b/be/src/olap/rowset/segment_v2/bitshuffle_page.h
@@ -93,13 +93,48 @@ public:
bool is_page_full() override { return _remain_element_capacity == 0; }
Status add(const uint8_t* vals, size_t* count) override {
+ return add_internal<false>(vals, count);
+ }
+
+ Status single_add(const uint8_t* vals, size_t* count) {
+ return add_internal<true>(vals, count);
+ }
+
+ template <bool single>
+ inline Status add_internal(const uint8_t* vals, size_t* count) {
DCHECK(!_finished);
+ if (_remain_element_capacity <= 0) {
+ *count = 0;
+ return Status::RuntimeError("page is full.");
+ }
int to_add = std::min<int>(_remain_element_capacity, *count);
- _data.append(vals, to_add * SIZE_OF_TYPE);
+ int to_add_size = to_add * SIZE_OF_TYPE;
+ size_t orig_size = _data.size();
+ _data.resize(orig_size + to_add_size);
_count += to_add;
_remain_element_capacity -= to_add;
// return added number through count
*count = to_add;
+ if constexpr (single) {
+ if constexpr (SIZE_OF_TYPE == 1) {
+ *reinterpret_cast<uint8_t*>(&_data[orig_size]) = *vals;
+ return Status::OK();
+ } else if constexpr (SIZE_OF_TYPE == 2) {
+ *reinterpret_cast<uint16_t*>(&_data[orig_size]) =
+ *reinterpret_cast<const uint16_t*>(vals);
+ return Status::OK();
+ } else if constexpr (SIZE_OF_TYPE == 4) {
+ *reinterpret_cast<uint32_t*>(&_data[orig_size]) =
+ *reinterpret_cast<const uint32_t*>(vals);
+ return Status::OK();
+ } else if constexpr (SIZE_OF_TYPE == 8) {
+ *reinterpret_cast<uint64_t*>(&_data[orig_size]) =
+ *reinterpret_cast<const uint64_t*>(vals);
+ return Status::OK();
+ }
+ }
+ // when single is true and SIZE_OF_TYPE > 8 or single is false
+ memcpy(&_data[orig_size], vals, to_add_size);
return Status::OK();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 03/05: [fix][feature](Function) fix return type && support hll_union_agg/group_concat agg to window function (#9119)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 355216f0ed5f7302f9e9444ca97a0ff58f3c052c
Author: zhangstar333 <87...@users.noreply.github.com>
AuthorDate: Sat May 7 20:44:04 2022 +0800
[fix][feature](Function) fix return type && support hll_union_agg/group_concat agg to window function (#9119)
---
.../vec/aggregate_functions/aggregate_function.h | 2 +-
.../aggregate_function_hll_union_agg.h | 4 +++
.../aggregate_function_stddev.h | 11 ++++--
.../java/org/apache/doris/catalog/FunctionSet.java | 8 ++---
.../data/correctness/test_select_variance_agg.out | 7 ++++
.../correctness/test_select_variance_agg.groovy | 41 ++++++++++++++++++++++
6 files changed, 66 insertions(+), 7 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function.h b/be/src/vec/aggregate_functions/aggregate_function.h
index 56e404d9e9..d822dada79 100644
--- a/be/src/vec/aggregate_functions/aggregate_function.h
+++ b/be/src/vec/aggregate_functions/aggregate_function.h
@@ -159,7 +159,7 @@ public:
}
}
//now this is use for sum/count/avg/min/max win function, other win function should override this function in class
- //stddev_pop/stddev_samp/variance_pop/variance_samp
+ //stddev_pop/stddev_samp/variance_pop/variance_samp/hll_union_agg/group_concat
void add_range_single_place(int64_t partition_start, int64_t partition_end, int64_t frame_start,
int64_t frame_end, AggregateDataPtr place, const IColumn** columns,
Arena* arena) const override {
diff --git a/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h b/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
index fe335b0509..c41e7be7d9 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
@@ -53,6 +53,8 @@ struct AggregateFunctionHLLData {
HyperLogLog get() const { return dst_hll; }
+ void reset() { dst_hll.clear(); }
+
void add(const IColumn* column, size_t row_num) {
if constexpr (is_nullable) {
auto* nullable_column = check_and_get_column<const ColumnNullable>(*column);
@@ -126,6 +128,8 @@ public:
Arena*) const override {
this->data(place).read(buf);
}
+
+ void reset(AggregateDataPtr __restrict place) const override { this->data(place).reset(); }
};
template <bool is_nullable = false>
diff --git a/be/src/vec/aggregate_functions/aggregate_function_stddev.h b/be/src/vec/aggregate_functions/aggregate_function_stddev.h
index d03774b26f..3531e8db49 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_stddev.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_stddev.h
@@ -21,6 +21,7 @@
#include "vec/aggregate_functions/aggregate_function.h"
#include "vec/columns/columns_number.h"
#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
#include "vec/data_types/data_type_number.h"
#include "vec/io/io_helper.h"
namespace doris::vectorized {
@@ -91,6 +92,8 @@ struct BaseData {
count += 1;
}
+ static DataTypePtr get_return_type() { return std::make_shared<DataTypeNumber<Float64>>(); }
+
double mean;
double m2;
int64_t count;
@@ -175,6 +178,10 @@ struct BaseDatadecimal {
count += 1;
}
+ static DataTypePtr get_return_type() {
+ return std::make_shared<DataTypeDecimal<Decimal128>>(27, 9);
+ }
+
DecimalV2Value mean;
DecimalV2Value m2;
int64_t count;
@@ -248,9 +255,9 @@ public:
DataTypePtr get_return_type() const override {
if constexpr (is_pop) {
- return std::make_shared<DataTypeFloat64>();
+ return Data::get_return_type();
} else {
- return make_nullable(std::make_shared<DataTypeFloat64>());
+ return make_nullable(Data::get_return_type());
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
index cc72786164..8096d14631 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
@@ -1620,7 +1620,7 @@ public class FunctionSet<min_initIN9doris_udf12DecimalV2ValEEEvPNS2_15FunctionCo
"_ZN5doris12HllFunctions9hll_mergeEPN9doris_udf15FunctionContextERKNS1_9StringValEPS4_",
"_ZN5doris12HllFunctions13hll_serializeEPN9doris_udf15FunctionContextERKNS1_9StringValE",
"_ZN5doris12HllFunctions12hll_finalizeEPN9doris_udf15FunctionContextERKNS1_9StringValE",
- true, false, true, true));
+ true, true, true, true));
// APPROX_COUNT_DISTINCT
// alias of ndv, compute approx count distinct use HyperLogLog
@@ -1641,7 +1641,7 @@ public class FunctionSet<min_initIN9doris_udf12DecimalV2ValEEEvPNS2_15FunctionCo
"_ZN5doris12HllFunctions9hll_mergeEPN9doris_udf15FunctionContextERKNS1_9StringValEPS4_",
"_ZN5doris12HllFunctions13hll_serializeEPN9doris_udf15FunctionContextERKNS1_9StringValE",
"_ZN5doris12HllFunctions12hll_finalizeEPN9doris_udf15FunctionContextERKNS1_9StringValE",
- true, false, true, true));
+ true, true, true, true));
// BITMAP_UNION_INT
addBuiltin(AggregateFunction.createBuiltin(BITMAP_UNION_INT,
@@ -2286,7 +2286,7 @@ public class FunctionSet<min_initIN9doris_udf12DecimalV2ValEEEvPNS2_15FunctionCo
prefix + "19string_concat_mergeEPN9doris_udf15FunctionContextERKNS1_9StringValEPS4_",
stringValSerializeOrFinalize,
prefix + "22string_concat_finalizeEPN9doris_udf15FunctionContextERKNS1_9StringValE", false,
- false, false, true));
+ true, false, true));
// Group_concat(string, string) vectorized
addBuiltin(AggregateFunction.createBuiltin("group_concat", Lists.<Type>newArrayList(Type.VARCHAR, Type.VARCHAR),
Type.VARCHAR, Type.VARCHAR, initNullString,
@@ -2294,7 +2294,7 @@ public class FunctionSet<min_initIN9doris_udf12DecimalV2ValEEEvPNS2_15FunctionCo
prefix + "19string_concat_mergeEPN9doris_udf15FunctionContextERKNS1_9StringValEPS4_",
stringValSerializeOrFinalize,
prefix + "22string_concat_finalizeEPN9doris_udf15FunctionContextERKNS1_9StringValE", false,
- false, false, true));
+ true, false, true));
// analytic functions
// Rank
diff --git a/regression-test/data/correctness/test_select_variance_agg.out b/regression-test/data/correctness/test_select_variance_agg.out
new file mode 100644
index 0000000000..66ad4d8c9e
--- /dev/null
+++ b/regression-test/data/correctness/test_select_variance_agg.out
@@ -0,0 +1,7 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select_default --
+31200.0
+
+-- !select_default2 --
+564.666666667
+
diff --git a/regression-test/suites/correctness/test_select_variance_agg.groovy b/regression-test/suites/correctness/test_select_variance_agg.groovy
new file mode 100644
index 0000000000..318679fac5
--- /dev/null
+++ b/regression-test/suites/correctness/test_select_variance_agg.groovy
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+ // or more contributor license agreements. See the NOTICE file
+ // distributed with this work for additional information
+ // regarding copyright ownership. The ASF licenses this file
+ // to you under the Apache License, Version 2.0 (the
+ // "License"); you may not use this file except in compliance
+ // with the License. You may obtain a copy of the License at
+ //
+ // http://www.apache.org/licenses/LICENSE-2.0
+ //
+ // Unless required by applicable law or agreed to in writing,
+ // software distributed under the License is distributed on an
+ // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ // KIND, either express or implied. See the License for the
+ // specific language governing permissions and limitations
+ // under the License.
+
+ suite("test_select_variance_agg") {
+ def tableName = "test_variance"
+
+
+ sql """ DROP TABLE IF EXISTS ${tableName} """
+ sql """
+ CREATE TABLE ${tableName} ( `aa` int NULL COMMENT "", `bb` decimal(27,9) NULL COMMENT "" )
+ ENGINE=OLAP UNIQUE KEY(`aa`) DISTRIBUTED BY HASH(`aa`) BUCKETS 3
+ PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" );
+ """
+
+ sql """ INSERT INTO ${tableName} VALUES
+ (123,34),
+ (423,78),
+ (3,23); """
+
+ // test_vectorized
+ sql """ set enable_vectorized_engine = true; """
+
+ qt_select_default """ select variance(aa) from ${tableName}; """
+
+ qt_select_default2 """ select variance(bb) from ${tableName}; """
+
+ }
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org