You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by li...@apache.org on 2022/01/17 12:31:41 UTC
[incubator-doris] branch vectorized updated (66b3b1d -> 8a1a612)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a change to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
omit 66b3b1d [Vectorized](compile) Fix compile error and warning (#7780)
omit c86d691 [Vectorized][Bug] Fix bug of repeated node resize and compile failed (#7778)
omit 778fa8d [Vectorized](improving) (exec) optimize VDataStreamSender's send() performance #7747 (#7751)
omit 57bdde6 [Vectorized][Improvement] Speed up column filtering via SIMD (#7775)
omit d2f2210 [Vectorized][feature](planner)(executor) Support grouping sets rollup cube (#7601)
omit 685f452 [Vectorized][Improvement] Enhancement unit test for vectorized function (#7750)
omit ee90c94 [Vectorization] Support SegmentIterator vectorization (#7613)
omit 24787ed [Vectorized][Function] Support function stddev/variance/stddev_samp/variance_samp (#7734)
omit fc05698 [Vectorized] Rebase code from master
omit e9056d6 [Vectorized][Bug] Bitmap/HLL type no support cast to varchar/char (#7737)
omit 2af5181 [Vectorized][Feature] upport function conv (#7693)
omit b79496b [Vectorized][Bug] Fix get wrong result when select random column && fix get wrong has_null_tag (#7728)
omit 28fb8c7 [Vectorized][Enhancement] use simd to speed up coalesce and if_not_null function (#7722)
omit 2c38a50 [Vectorized][Enhancement] fix some bug & improve some code (#7714)
omit 27d3898 [Vectorized][Bug] fix 'negative' function ut run fail && fix testIsBucketShuffleJoin run fail && fix some compile fail (#7688)
omit 3e45025 [Vectorized] (olap) Optimize BlockReader's performance (#7642)
omit 0dd1662 [Feature][Vectorized] Support String in vec exe engine (#7670)
omit a051b33 [Vectorized] [Function] Support do not fold constant at vectorized (#7668)
omit 952f0e3 [Vectorized] Support bloom filter predicate on vectorized engine storage layer (#7557)
omit 77e0212 [vectorized] [block] Add new method get_data_type to avoid unnecessary copy by the method get_data_type (#7600)
omit 01d9434 [Vectorized][Feature] support money_format/ucase/character_length (#7649)
omit 9432587 [Vectorizd] [Function] Add string type vec support at doris_builtins_functions[D (#7661)
omit ead467c [Bug] Fix function nulllable not match and largetint cast failed (#7659)
omit 3425e8a [Function][Vec] add function coalesce (#7632)
omit bdeb6b7 [Vectorized][Feature] fix core dump when using function override and function alias at the same time && support substr(str,int) override (#7640)
omit c4623f2 [Bug] Fix bug of cast expr nullable and ifnull function (#7626)
omit fb945cd [Refactor] Cow refactor: giveup using boost (#7567)
omit 326f0d7 [Vectorized][Function] Support function and (#7618)
omit 3339878 [Bug] Change parser string to int (#7595)
omit 2d31421 [Bug] Fix bug of concat function and fold const expr (#7608)
omit 204a35d [Function] Fix error about rank/dense_rank/row_number return always not nullable (#7561)
omit 54bd985 [Bug] Fix negative function error result and sort node eos (#7555)
omit 07abe49 [Vectorized Exec Engine] Support Vectorized Exec Engine In Doris
add b51121f [chore](github-action) Add label auto for pull requests (#7663)
add d1a994e [fix](cpu-resource)(resource-tag) Allow set cpu_resource_limit to -1 and fix resource tag bug(#6830)
add 3da4425 [fix](github-action) fix the action of set-label-based-on-pr-title (#7757)
add 10709f3 [fix](github-action) fix the action of set-label-based-on-pr-title (#7758)
add d03151b [chore](be) Add -Werror (#7744)
add 902ab93 [fix](session-variable) fix bug that checkpoint may overwrite the global variables (#7526)
add 6188ab2 [docs](faq) add multiple FE WEB UI login issues (#7654)
add f381782 [fix] fix malloc and free mismatch issue (#7702)
add fe80d14 [style] replace Chinese comments with English comments (#7732)
add 5c4055a [style] Translate Chinese to English in be_olap_field.h (#7738)
add e7d65e4 [style] translate code annotations into english (#7752)
add a6ff1bd Flink / Spark connector compilation problem (#7725)
add be43316 [docs] add doc for community feedback and fix CI (#7759)
add 4a3cbf5 [fix](show-load) fix show load with the same column name in Where Clause (#7523)
add 5b0f11b [feature](mysql-compatibility)(function) add `WEEKDAY` function (#7673)
add 8b7d7e4 [improvement] create/drop index support if [not] exist (#7748)
add 5f8d912 [improvement](routine-load) Reduce the probability that the routine load task rpc timeout (#7754)
add 36d6d23 [refactor] remove duplicate if that will never be used (#7761)
add 88a3d08 [fix] fix NPE in SysVariableDesc::equal (#7766)
add 5c7863c [improvement](fe-unit-test) Fix port in use when the cluster starts in UT. (#7768)
add e80c34b [docs][typo] fix some typos in documents (#7769)
add ebc27a4 [docs] Split the FAQ And Revert auto-label action (#7770)
new 14d84ba [Vectorized Exec Engine] Support Vectorized Exec Engine In Doris
new a33b2ac [Bug] Fix negative function error result and sort node eos (#7555)
new 9431a93 [Function] Fix error about rank/dense_rank/row_number return always not nullable (#7561)
new 5f52c04 [Bug] Fix bug of concat function and fold const expr (#7608)
new 4528643 [Bug] Change parser string to int (#7595)
new 5779e9b [Vectorized][Function] Support function and (#7618)
new a261538 [Refactor] Cow refactor: giveup using boost (#7567)
new 24e1d64 [Bug] Fix bug of cast expr nullable and ifnull function (#7626)
new 41b165c [Vectorized][Feature] fix core dump when using function override and function alias at the same time && support substr(str,int) override (#7640)
new ebdc0ba [Function][Vec] add function coalesce (#7632)
new da43e38 [Bug] Fix function nulllable not match and largetint cast failed (#7659)
new 345c119 [Vectorizd] [Function] Add string type vec support at doris_builtins_functions[D (#7661)
new cc451e7 [Vectorized][Feature] support money_format/ucase/character_length (#7649)
new e4619d9 [vectorized] [block] Add new method get_data_type to avoid unnecessary copy by the method get_data_type (#7600)
new ae9d0cf [Vectorized] Support bloom filter predicate on vectorized engine storage layer (#7557)
new acb63c7 [Vectorized] [Function] Support do not fold constant at vectorized (#7668)
new 1fc1c70 [Feature][Vectorized] Support String in vec exe engine (#7670)
new 8b82104 [Vectorized] (olap) Optimize BlockReader's performance (#7642)
new 9f16ac2 [Vectorized][Bug] fix 'negative' function ut run fail && fix testIsBucketShuffleJoin run fail && fix some compile fail (#7688)
new b20b5b7 [Vectorized][Enhancement] fix some bug & improve some code (#7714)
new f3ce1cb [Vectorized][Enhancement] use simd to speed up coalesce and if_not_null function (#7722)
new a9d9c02 [Vectorized][Bug] Fix get wrong result when select random column && fix get wrong has_null_tag (#7728)
new ffdc9fc [Vectorized][Feature] upport function conv (#7693)
new 30de672 [Vectorized][Bug] Bitmap/HLL type no support cast to varchar/char (#7737)
new 20619e7 [Vectorized] Rebase code from master
new 990723c [Vectorized][Function] Support function stddev/variance/stddev_samp/variance_samp (#7734)
new e23e332 [Vectorization] Support SegmentIterator vectorization (#7613)
new d1c75a0 [Vectorized][Improvement] Enhancement unit test for vectorized function (#7750)
new 8760e64 [Vectorized][feature](planner)(executor) Support grouping sets rollup cube (#7601)
new 0a68fc3 [Vectorized][Improvement] Speed up column filtering via SIMD (#7775)
new f45d2a2 [Vectorized](improving) (exec) optimize VDataStreamSender's send() performance #7747 (#7751)
new b634ea6 [Vectorized][Bug] Fix bug of repeated node resize and compile failed (#7778)
new 8a1a612 [Vectorized](compile) Fix compile error and warning (#7780)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (66b3b1d)
\
N -- N -- N refs/heads/vectorized (8a1a612)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 33 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/CMakeLists.txt | 9 +-
be/src/common/config.h | 20 ++
be/src/common/daemon.cpp | 10 +-
be/src/exec/csv_scan_node.cpp | 7 -
be/src/exec/s3_reader.cpp | 5 +-
be/src/exec/s3_writer.cpp | 14 +-
be/src/exec/tablet_sink.cpp | 11 +-
be/src/exec/tablet_sink.h | 2 +-
be/src/exprs/timestamp_functions.cpp | 11 ++
be/src/exprs/timestamp_functions.h | 2 +
be/src/olap/compress.h | 24 +--
be/src/olap/delta_writer.cpp | 3 +-
be/src/olap/delta_writer.h | 1 +
be/src/olap/field.h | 22 ++-
be/src/olap/memtable_flush_executor.cpp | 29 ++-
be/src/olap/memtable_flush_executor.h | 8 +-
be/src/olap/row_cursor.h | 29 ++-
be/src/olap/wrapper_field.h | 2 +-
be/src/runtime/load_channel.cpp | 12 +-
be/src/runtime/load_channel.h | 13 +-
be/src/runtime/load_channel_mgr.cpp | 16 +-
be/src/runtime/routine_load/data_consumer.cpp | 2 +
.../routine_load/routine_load_task_executor.cpp | 2 +-
be/src/runtime/tablets_channel.cpp | 6 +-
be/src/runtime/tablets_channel.h | 4 +-
be/test/exprs/timestamp_functions_test.cpp | 27 +++
docs/.vuepress/sidebar/en.js | 130 ++++++-------
docs/.vuepress/sidebar/zh-CN.js | 118 ++++++------
docs/en/administrator-guide/config/be_config.md | 27 +++
.../operation/metadata-operation.md | 2 +-
docs/en/community/feedback.md | 22 ++-
docs/en/faq/error.md | 149 +++++++++++++++
docs/en/faq/faq.md | 209 ++++++++------------
.../sql-functions/date-time-functions/dayofweek.md | 2 +-
.../weekday.md} | 51 +++--
.../sql-statements/Administration/SHOW PLUGINS.md | 2 +-
.../sql-statements/Data Definition/ALTER TABLE.md | 8 +-
.../sql-statements/Data Definition/CREATE INDEX.md | 4 +-
.../Data Definition/CREATE REPOSITORY.md | 2 +-
.../sql-statements/Data Definition/DROP INDEX.md | 2 +-
.../Data Definition/DROP REPOSITORY.md | 2 +-
.../Data Definition/SHOW RESOURCES.md | 2 +-
.../Data Manipulation/lateral-view.md | 2 +-
docs/zh-CN/administrator-guide/config/be_config.md | 24 +++
.../operation/metadata-operation.md | 4 +-
docs/zh-CN/community/feedback.md | 24 ++-
docs/zh-CN/faq/error.md | 147 ++++++++++++++
docs/zh-CN/faq/faq.md | 208 ++++++++------------
.../sql-functions/date-time-functions/dayofweek.md | 2 +-
.../weekday.md} | 54 +++---
.../sql-statements/Administration/SHOW PLUGINS.md | 2 +-
.../sql-statements/Data Definition/ALTER TABLE.md | 8 +-
.../sql-statements/Data Definition/CREATE INDEX.md | 4 +-
.../Data Definition/CREATE REPOSITORY.md | 2 +-
.../Data Definition/CREATE RESOURCE.md | 2 +-
.../sql-statements/Data Definition/DROP INDEX.md | 2 +-
.../Data Definition/DROP REPOSITORY.md | 2 +-
.../Data Definition/DROP RESOURCE.md | 2 +-
.../Data Manipulation/lateral-view.md | 2 +-
extension/flink-doris-connector/build.sh | 8 +-
extension/flink-doris-connector/pom.xml | 26 +--
extension/spark-doris-connector/build.sh | 8 +-
extension/spark-doris-connector/pom.xml | 26 +--
fe/fe-core/src/main/cup/sql_parser.cup | 16 +-
.../apache/doris/alter/SchemaChangeHandler.java | 32 +++-
.../org/apache/doris/analysis/DropIndexClause.java | 9 +-
.../java/org/apache/doris/analysis/IndexDef.java | 8 +-
.../org/apache/doris/analysis/ShowLoadStmt.java | 10 +
.../org/apache/doris/analysis/SysVariableDesc.java | 5 +-
.../org/apache/doris/catalog/AccessPrivilege.java | 4 +-
.../java/org/apache/doris/catalog/Catalog.java | 10 +-
.../org/apache/doris/clone/TabletScheduler.java | 15 ++
.../java/org/apache/doris/common/ErrorCode.java | 4 +-
.../org/apache/doris/journal/JournalEntity.java | 9 +-
.../java/org/apache/doris/master/Checkpoint.java | 34 +++-
.../java/org/apache/doris/mysql/MysqlProto.java | 2 +-
.../apache/doris/mysql/privilege/UserProperty.java | 9 +-
.../java/org/apache/doris/persist/EditLog.java | 20 +-
.../java/org/apache/doris/qe/ConnectProcessor.java | 2 +-
.../main/java/org/apache/doris/qe/VariableMgr.java | 138 ++++++++------
.../doris/transaction/GlobalTransactionMgr.java | 44 ++---
.../doris/analysis/CreateIndexClauseTest.java | 2 +-
.../apache/doris/analysis/DropIndexClauseTest.java | 5 +-
.../org/apache/doris/analysis/IndexDefTest.java | 6 +-
.../apache/doris/analysis/ShowLoadStmtTest.java | 41 +++-
.../org/apache/doris/catalog/UserPropertyTest.java | 24 ++-
.../java/org/apache/doris/qe/VariableMgrTest.java | 211 +++++++++++----------
.../org/apache/doris/utframe/UtFrameUtils.java | 28 +--
gensrc/proto/internal_service.proto | 3 +
gensrc/script/doris_builtins_functions.py | 3 +
90 files changed, 1394 insertions(+), 882 deletions(-)
create mode 100644 docs/en/faq/error.md
copy docs/en/sql-reference/sql-functions/{string-functions/json_object.md => date-time-functions/weekday.md} (54%)
create mode 100644 docs/zh-CN/faq/error.md
copy docs/zh-CN/sql-reference/sql-functions/{string-functions/json_object.md => date-time-functions/weekday.md} (53%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 33/33: [Vectorized](compile) Fix compile error and warning (#7780)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8a1a6126b4c387eaa678f4a65e773d795e021f0a
Author: Zeno Yang <co...@qq.com>
AuthorDate: Mon Jan 17 20:27:13 2022 +0800
[Vectorized](compile) Fix compile error and warning (#7780)
---
be/src/vec/columns/column.h | 1 +
be/src/vec/functions/function.h | 3 +++
be/src/vec/functions/function_grouping.cpp | 4 ++--
3 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index d58979d..71b86ae 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -220,6 +220,7 @@ public:
*/
virtual Ptr filter_by_selector(const uint16_t* sel, size_t sel_size, Ptr* ptr = nullptr) {
LOG(FATAL) << "column not support filter_by_selector";
+ __builtin_unreachable();
};
/// Permutes elements using specified permutation. Is used in sortings.
diff --git a/be/src/vec/functions/function.h b/be/src/vec/functions/function.h
index bc76a10..cf00c08 100644
--- a/be/src/vec/functions/function.h
+++ b/be/src/vec/functions/function.h
@@ -404,6 +404,7 @@ public:
const ColumnNumbers& /*arguments*/,
size_t /*result*/) const final {
LOG(FATAL) << "prepare is not implemented for IFunction";
+ __builtin_unreachable();
}
Status prepare(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
@@ -412,10 +413,12 @@ public:
[[noreturn]] const DataTypes& get_argument_types() const final {
LOG(FATAL) << "get_argument_types is not implemented for IFunction";
+ __builtin_unreachable();
}
[[noreturn]] const DataTypePtr& get_return_type() const final {
LOG(FATAL) << "get_return_type is not implemented for IFunction";
+ __builtin_unreachable();
}
protected:
diff --git a/be/src/vec/functions/function_grouping.cpp b/be/src/vec/functions/function_grouping.cpp
index 763dec2..07872ee 100644
--- a/be/src/vec/functions/function_grouping.cpp
+++ b/be/src/vec/functions/function_grouping.cpp
@@ -15,11 +15,11 @@
// specific language governing permissions and limitations
// under the License.
-#include "function_grouping.h"
+#include "vec/functions/function_grouping.h"
namespace doris::vectorized {
void register_function_grouping(SimpleFunctionFactory& factory) {
factory.register_function<FunctionGrouping>();
factory.register_function<FunctionGroupingId>();
}
-}
+} // namespace doris::vectorized
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 15/33: [Vectorized] Support bloom filter predicate on vectorized engine storage layer (#7557)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit ae9d0cff0ad6aa59919eaa138307d619fffa9aeb
Author: Zeno Yang <co...@qq.com>
AuthorDate: Sat Jan 8 01:00:43 2022 +0800
[Vectorized] Support bloom filter predicate on vectorized engine storage layer (#7557)
---
be/src/olap/bloom_filter_predicate.h | 40 +++++-
.../olap/bloom_filter_column_predicate_test.cpp | 36 ++++++
be/test/olap/null_predicate_test.cpp | 144 +++++++++++++++++++++
3 files changed, 219 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/bloom_filter_predicate.h b/be/src/olap/bloom_filter_predicate.h
index b3dcbbb..ff3201c 100644
--- a/be/src/olap/bloom_filter_predicate.h
+++ b/be/src/olap/bloom_filter_predicate.h
@@ -27,6 +27,10 @@
#include "olap/field.h"
#include "runtime/string_value.hpp"
#include "runtime/vectorized_row_batch.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/predicate_column.h"
+#include "vec/utils/util.hpp"
namespace doris {
@@ -59,12 +63,14 @@ public:
return Status::OK();
}
+ void evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const override;
+
private:
std::shared_ptr<IBloomFilterFuncBase> _filter;
SpecificFilter* _specific_filter; // owned by _filter
};
-// blomm filter column predicate do not support in segment v1
+// bloom filter column predicate do not support in segment v1
template <PrimitiveType type>
void BloomFilterColumnPredicate<type>::evaluate(VectorizedRowBatch* batch) const {
uint16_t n = batch->size();
@@ -99,6 +105,38 @@ void BloomFilterColumnPredicate<type>::evaluate(ColumnBlock* block, uint16_t* se
*size = new_size;
}
+template <PrimitiveType type>
+void BloomFilterColumnPredicate<type>::evaluate(vectorized::IColumn& column, uint16_t* sel,
+ uint16_t* size) const {
+ uint16_t new_size = 0;
+ using T = typename PrimitiveTypeTraits<type>::CppType;
+
+ if (column.is_nullable()) {
+ auto* nullable_col = vectorized::check_and_get_column<vectorized::ColumnNullable>(column);
+ auto& null_map_data = nullable_col->get_null_map_column().get_data();
+ auto* pred_col = vectorized::check_and_get_column<vectorized::PredicateColumnType<T>>(
+ nullable_col->get_nested_column());
+ auto& pred_col_data = pred_col->get_data();
+ for (uint16_t i = 0; i < *size; i++) {
+ uint16_t idx = sel[i];
+ sel[new_size] = idx;
+ const auto* cell_value = reinterpret_cast<const void*>(&(pred_col_data[idx]));
+ new_size += (!null_map_data[idx]) && _specific_filter->find_olap_engine(cell_value);
+ }
+ } else {
+ auto* pred_col =
+ vectorized::check_and_get_column<vectorized::PredicateColumnType<T>>(column);
+ auto& pred_col_data = pred_col->get_data();
+ for (uint16_t i = 0; i < *size; i++) {
+ uint16_t idx = sel[i];
+ sel[new_size] = idx;
+ const auto* cell_value = reinterpret_cast<const void*>(&(pred_col_data[idx]));
+ new_size += _specific_filter->find_olap_engine(cell_value);
+ }
+ }
+ *size = new_size;
+}
+
class BloomFilterColumnPredicateFactory {
public:
static ColumnPredicate* create_column_predicate(
diff --git a/be/test/olap/bloom_filter_column_predicate_test.cpp b/be/test/olap/bloom_filter_column_predicate_test.cpp
index 164c51d..24abea1 100644
--- a/be/test/olap/bloom_filter_column_predicate_test.cpp
+++ b/be/test/olap/bloom_filter_column_predicate_test.cpp
@@ -28,6 +28,11 @@
#include "runtime/string_value.hpp"
#include "runtime/vectorized_row_batch.h"
#include "util/logging.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/predicate_column.h"
+#include "vec/core/block.h"
+
+using namespace doris::vectorized;
namespace doris {
@@ -172,6 +177,37 @@ TEST_F(TestBloomFilterColumnPredicate, FLOAT_COLUMN) {
ASSERT_EQ(select_size, 1);
ASSERT_FLOAT_EQ(*(float*)col_block.cell(_row_block->selection_vector()[0]).cell_ptr(), 5.1);
+ // for vectorized::Block no null
+ auto pred_col = PredicateColumnType<vectorized::Float32>::create();
+ pred_col->reserve(size);
+ for (int i = 0; i < size; ++i) {
+ *(col_data + i) = i + 0.1f;
+ pred_col->insert_data(reinterpret_cast<const char*>(col_data + i), 0);
+ }
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ pred->evaluate(*pred_col, _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 3);
+ ASSERT_FLOAT_EQ((float)pred_col->get_data()[_row_block->selection_vector()[0]], 4.1);
+ ASSERT_FLOAT_EQ((float)pred_col->get_data()[_row_block->selection_vector()[1]], 5.1);
+ ASSERT_FLOAT_EQ((float)pred_col->get_data()[_row_block->selection_vector()[2]], 6.1);
+
+ // for vectorized::Block has nulls
+ auto null_map = ColumnUInt8::create(size, 0);
+ auto& null_map_data = null_map->get_data();
+ for (int i = 0; i < size; ++i) {
+ null_map_data[i] = (i % 2 == 0);
+ }
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ auto nullable_col =
+ vectorized::ColumnNullable::create(std::move(pred_col), std::move(null_map));
+ pred->evaluate(*nullable_col, _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 1);
+ auto nested_col = check_and_get_column<PredicateColumnType<vectorized::Float32>>(
+ nullable_col->get_nested_column());
+ ASSERT_FLOAT_EQ((float)nested_col->get_data()[_row_block->selection_vector()[0]], 5.1);
+
delete pred;
}
diff --git a/be/test/olap/null_predicate_test.cpp b/be/test/olap/null_predicate_test.cpp
index bf53ec2..ec870af 100644
--- a/be/test/olap/null_predicate_test.cpp
+++ b/be/test/olap/null_predicate_test.cpp
@@ -28,6 +28,10 @@
#include "runtime/string_value.hpp"
#include "runtime/vectorized_row_batch.h"
#include "util/logging.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/core/block.h"
+
+using namespace doris::vectorized;
namespace doris {
@@ -142,6 +146,16 @@ public:
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size); \
ASSERT_EQ(select_size, 0); \
\
+ /* for vectorized::Block no null */ \
+ _row_block->clear(); \
+ select_size = _row_block->selected_size(); \
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns); \
+ _row_block->convert_to_vec_block(&vec_block); \
+ ColumnPtr vec_col = vec_block.get_columns()[0]; \
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col), \
+ _row_block->selection_vector(), &select_size); \
+ ASSERT_EQ(select_size, 0); \
+ \
/* for has nulls */ \
col_vector->set_no_nulls(false); \
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size)); \
@@ -173,6 +187,16 @@ public:
select_size = _row_block->selected_size(); \
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size); \
ASSERT_EQ(select_size, 5); \
+ \
+ /* for vectorized::Block has nulls */ \
+ _row_block->clear(); \
+ select_size = _row_block->selected_size(); \
+ vec_block = tablet_schema.create_block(return_columns); \
+ _row_block->convert_to_vec_block(&vec_block); \
+ vec_col = vec_block.get_columns()[0]; \
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col), \
+ _row_block->selection_vector(), &select_size); \
+ ASSERT_EQ(select_size, 5); \
pred.reset(); \
}
@@ -216,6 +240,16 @@ TEST_F(TestNullPredicate, FLOAT_COLUMN) {
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 0);
+ // for vectorized::Block no null
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ ColumnPtr vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 0);
+
// for VectorizedBatch has nulls
col_vector->set_no_nulls(false);
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size));
@@ -247,6 +281,16 @@ TEST_F(TestNullPredicate, FLOAT_COLUMN) {
select_size = _row_block->selected_size();
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 5);
+
+ // for vectorized::Block has nulls
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 5);
}
TEST_F(TestNullPredicate, DOUBLE_COLUMN) {
@@ -284,6 +328,16 @@ TEST_F(TestNullPredicate, DOUBLE_COLUMN) {
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 0);
+ // for vectorized::Block no null
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ ColumnPtr vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 0);
+
// for VectorizedBatch has nulls
col_vector->set_no_nulls(false);
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size));
@@ -315,6 +369,16 @@ TEST_F(TestNullPredicate, DOUBLE_COLUMN) {
select_size = _row_block->selected_size();
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 5);
+
+ // for vectorized::Block has nulls
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 5);
}
TEST_F(TestNullPredicate, DECIMAL_COLUMN) {
@@ -355,6 +419,16 @@ TEST_F(TestNullPredicate, DECIMAL_COLUMN) {
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 0);
+ // for vectorized::Block no null
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ ColumnPtr vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 0);
+
// for VectorizedBatch has nulls
col_vector->set_no_nulls(false);
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size));
@@ -388,6 +462,16 @@ TEST_F(TestNullPredicate, DECIMAL_COLUMN) {
select_size = _row_block->selected_size();
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 4);
+
+ // for vectorized::Block has nulls
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 4);
}
TEST_F(TestNullPredicate, STRING_COLUMN) {
@@ -440,6 +524,16 @@ TEST_F(TestNullPredicate, STRING_COLUMN) {
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 0);
+ // for vectorized::Block no null
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ ColumnPtr vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 0);
+
// for VectorizedBatch has nulls
col_vector->set_no_nulls(false);
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size));
@@ -483,6 +577,16 @@ TEST_F(TestNullPredicate, STRING_COLUMN) {
select_size = _row_block->selected_size();
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 4);
+
+ // for vectorized::Block has nulls
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 4);
}
TEST_F(TestNullPredicate, DATE_COLUMN) {
@@ -529,6 +633,16 @@ TEST_F(TestNullPredicate, DATE_COLUMN) {
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 0);
+ // for vectorized::Block no null
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ ColumnPtr vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 0);
+
// for VectorizedBatch has nulls
col_vector->set_no_nulls(false);
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size));
@@ -562,6 +676,16 @@ TEST_F(TestNullPredicate, DATE_COLUMN) {
select_size = _row_block->selected_size();
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 2);
+
+ // for vectorized::Block has nulls
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 2);
}
TEST_F(TestNullPredicate, DATETIME_COLUMN) {
@@ -608,6 +732,16 @@ TEST_F(TestNullPredicate, DATETIME_COLUMN) {
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 0);
+ // for vectorized::Block no null
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vectorized::Block vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ ColumnPtr vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 0);
+
// for VectorizedBatch has nulls
col_vector->set_no_nulls(false);
bool* is_null = reinterpret_cast<bool*>(_mem_pool->allocate(size));
@@ -641,6 +775,16 @@ TEST_F(TestNullPredicate, DATETIME_COLUMN) {
select_size = _row_block->selected_size();
pred->evaluate(&col_block, _row_block->selection_vector(), &select_size);
ASSERT_EQ(select_size, 2);
+
+ // for vectorized::Block has nulls
+ _row_block->clear();
+ select_size = _row_block->selected_size();
+ vec_block = tablet_schema.create_block(return_columns);
+ _row_block->convert_to_vec_block(&vec_block);
+ vec_col = vec_block.get_columns()[0];
+ pred->evaluate(const_cast<doris::vectorized::IColumn&>(*vec_col),
+ _row_block->selection_vector(), &select_size);
+ ASSERT_EQ(select_size, 2);
}
} // namespace doris
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 07/33: [Refactor] Cow refactor: giveup using boost (#7567)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a261538948f61f9b38c4e4d906bd9254bbae07b4
Author: thinker <zc...@qq.com>
AuthorDate: Thu Jan 6 00:12:17 2022 +0800
[Refactor] Cow refactor: giveup using boost (#7567)
Co-authored-by: zuochunwei <zu...@meituan.com>
---
be/src/vec/common/cow.h | 172 ++++++++++++++++++++++++++------
be/src/vec/exec/vanalytic_eval_node.cpp | 2 +-
be/src/vec/functions/function_cast.h | 2 +-
3 files changed, 146 insertions(+), 30 deletions(-)
diff --git a/be/src/vec/common/cow.h b/be/src/vec/common/cow.h
index dbee2b9..08edb89 100644
--- a/be/src/vec/common/cow.h
+++ b/be/src/vec/common/cow.h
@@ -24,6 +24,7 @@
#include <boost/smart_ptr/intrusive_ref_counter.hpp>
#include <initializer_list>
+
/** Copy-on-write shared ptr.
* Allows to work with shared immutable objects and sometimes unshare and mutate you own unique copy.
*
@@ -92,36 +93,158 @@
* to use std::unique_ptr for it somehow.
*/
template <typename Derived>
-class COW : public boost::intrusive_ref_counter<Derived> {
-private:
+class COW {
+ std::atomic_uint ref_counter;
+
+protected:
+ COW() : ref_counter(0) {}
+
+ COW(COW const&) : ref_counter(0) {}
+
+ COW& operator=(COW const&) {
+ return *this;
+ }
+
+ unsigned int use_count() const {
+ return ref_counter.load();
+ }
+
+ void add_ref() {
+ ++ref_counter;
+ }
+
+ void release_ref() {
+ if (--ref_counter == 0) {
+ delete static_cast<const Derived*>(this);
+ }
+ }
+
Derived* derived() { return static_cast<Derived*>(this); }
+
const Derived* derived() const { return static_cast<const Derived*>(this); }
template <typename T>
- class IntrusivePtr : public boost::intrusive_ptr<T> {
+ class intrusive_ptr {
public:
- using boost::intrusive_ptr<T>::intrusive_ptr;
+ intrusive_ptr() : t(nullptr) {}
+
+ intrusive_ptr(T* t, bool add_ref=true) : t(t) {
+ if (t && add_ref) ((std::remove_const_t<T>*)t)->add_ref();
+ }
+
+ template <typename U>
+ intrusive_ptr(intrusive_ptr<U> const& rhs) : t(rhs.get()) {
+ if (t) ((std::remove_const_t<T>*)t)->add_ref();
+ }
+
+ intrusive_ptr(intrusive_ptr const& rhs) : t(rhs.get()) {
+ if (t) ((std::remove_const_t<T>*)t)->add_ref();
+ }
+
+ ~intrusive_ptr() {
+ if (t) ((std::remove_const_t<T>*)t)->release_ref();
+ }
+
+ template <typename U>
+ intrusive_ptr& operator=(intrusive_ptr<U> const& rhs) {
+ intrusive_ptr(rhs).swap(*this);
+ return *this;
+ }
+
+ intrusive_ptr(intrusive_ptr&& rhs) : t(rhs.t) {
+ rhs.t = nullptr;
+ }
+
+ intrusive_ptr& operator=(intrusive_ptr&& rhs) {
+ intrusive_ptr(static_cast<intrusive_ptr&&>(rhs)).swap(*this);
+ return *this;
+ }
+
+ template<class U> friend class intrusive_ptr;
+
+ template<class U>
+ intrusive_ptr(intrusive_ptr<U>&& rhs) : t(rhs.t) {
+ rhs.t = nullptr;
+ }
+
+ template<class U>
+ intrusive_ptr& operator=(intrusive_ptr<U>&& rhs) {
+ intrusive_ptr(static_cast<intrusive_ptr<U>&&>(rhs)).swap(*this);
+ return *this;
+ }
+
+ intrusive_ptr& operator=(intrusive_ptr const& rhs) {
+ intrusive_ptr(rhs).swap(*this);
+ return *this;
+ }
+
+ intrusive_ptr& operator=(T* rhs) {
+ intrusive_ptr(rhs).swap(*this);
+ return *this;
+ }
+
+ void reset() {
+ intrusive_ptr().swap(*this);
+ }
+
+ void reset(T* rhs) {
+ intrusive_ptr(rhs).swap(*this);
+ }
+
+ void reset(T* rhs, bool add_ref) {
+ intrusive_ptr(rhs, add_ref).swap(*this);
+ }
+
+ T* get() const {
+ return t;
+ }
+
+ T* detach() {
+ T* ret = t;
+ t = nullptr;
+ return ret;
+ }
+
+ void swap(intrusive_ptr& rhs) {
+ T* tmp = t;
+ t = rhs.t;
+ rhs.t = tmp;
+ }
+
+ T& operator*() const& {
+ return *t;
+ }
- T& operator*() const& { return boost::intrusive_ptr<T>::operator*(); }
T&& operator*() const&& {
- return const_cast<typename std::remove_const<T>::type&&>(
- *boost::intrusive_ptr<T>::get());
+ return const_cast<std::remove_const_t<T>&&>(*t);
+ }
+
+ T* operator->() const {
+ return t;
+ }
+
+ operator bool() const {
+ return t != nullptr;
+ }
+
+ operator T*() const {
+ return t;
}
+
+ private:
+ T* t;
};
protected:
template <typename T>
- class mutable_ptr : public IntrusivePtr<T> {
+ class mutable_ptr : public intrusive_ptr<T> {
private:
- using Base = IntrusivePtr<T>;
+ using Base = intrusive_ptr<T>;
- template <typename>
- friend class COW;
- template <typename, typename>
- friend class COWHelper;
+ template <typename> friend class COW;
+ template <typename, typename> friend class COWHelper;
explicit mutable_ptr(T* ptr) : Base(ptr) {}
-
public:
/// Copy: not possible.
mutable_ptr(const mutable_ptr&) = delete;
@@ -144,17 +267,14 @@ public:
protected:
template <typename T>
- class immutable_ptr : public IntrusivePtr<const T> {
+ class immutable_ptr : public intrusive_ptr<const T> {
private:
- using Base = IntrusivePtr<const T>;
+ using Base = intrusive_ptr<const T>;
- template <typename>
- friend class COW;
- template <typename, typename>
- friend class COWHelper;
+ template <typename> friend class COW;
+ template <typename, typename> friend class COWHelper;
explicit immutable_ptr(const T* ptr) : Base(ptr) {}
-
public:
/// Copy from immutable ptr: ok.
immutable_ptr(const immutable_ptr&) = default;
@@ -198,8 +318,8 @@ public:
}
public:
- Ptr get_ptr() const { return static_cast<Ptr>(derived()); }
- MutablePtr get_ptr() { return static_cast<MutablePtr>(derived()); }
+ Ptr get_ptr() const { return Ptr(derived()); }
+ MutablePtr get_ptr() { return MutablePtr(derived()); }
protected:
MutablePtr shallow_mutate() const {
@@ -294,10 +414,6 @@ public:
*/
template <typename Base, typename Derived>
class COWHelper : public Base {
-private:
- Derived* derived() { return static_cast<Derived*>(this); }
- const Derived* derived() const { return static_cast<const Derived*>(this); }
-
public:
using Ptr = typename Base::template immutable_ptr<Derived>;
using MutablePtr = typename Base::template mutable_ptr<Derived>;
@@ -313,7 +429,7 @@ public:
}
typename Base::MutablePtr clone() const override {
- return typename Base::MutablePtr(new Derived(*derived()));
+ return typename Base::MutablePtr(new Derived(static_cast<const Derived&>(*this)));
}
protected:
diff --git a/be/src/vec/exec/vanalytic_eval_node.cpp b/be/src/vec/exec/vanalytic_eval_node.cpp
index e828a73..4d69716 100644
--- a/be/src/vec/exec/vanalytic_eval_node.cpp
+++ b/be/src/vec/exec/vanalytic_eval_node.cpp
@@ -391,7 +391,7 @@ BlockRowPos VAnalyticEvalNode::_compare_row_to_find_end(int idx, BlockRowPos sta
}
//check whether need get column again, maybe same as first init
- if (start_column != start_next_block_column) {
+ if (start_column.get() != start_next_block_column.get()) {
start_init_row_num = 0;
start.block_num = start_block_num;
start_column = _input_blocks[start.block_num].get_by_position(idx).column;
diff --git a/be/src/vec/functions/function_cast.h b/be/src/vec/functions/function_cast.h
index 03856b9..0aa93d0 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -1110,7 +1110,7 @@ private:
const auto& tmp_res = tmp_block.get_by_position(tmp_res_index);
/// May happen in fuzzy tests. For debug purpose.
- if (!tmp_res.column) {
+ if (!tmp_res.column.get()) {
return Status::RuntimeError(fmt::format(
"Couldn't convert {} to {} in prepare_remove_nullable wrapper.",
block.get_by_position(arguments[0]).type->get_name(),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 13/33: [Vectorized][Feature] support money_format/ucase/character_length (#7649)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit cc451e7126235327872f1d043ce41d09460fbd79
Author: Pxl <95...@qq.com>
AuthorDate: Fri Jan 7 15:04:55 2022 +0800
[Vectorized][Feature] support money_format/ucase/character_length (#7649)
---
be/src/vec/functions/function_string.cpp | 12 +++-
be/src/vec/functions/function_string.h | 110 +++++++++++++++++++++++++++++-
gensrc/script/doris_builtins_functions.py | 8 +--
3 files changed, 120 insertions(+), 10 deletions(-)
diff --git a/be/src/vec/functions/function_string.cpp b/be/src/vec/functions/function_string.cpp
index 34f1c6b..bdeb7e5 100644
--- a/be/src/vec/functions/function_string.cpp
+++ b/be/src/vec/functions/function_string.cpp
@@ -272,7 +272,7 @@ struct HexStringName {
struct HexStringImpl {
static DataTypes get_variadic_argument_types() {
- return {std::make_shared<vectorized::DataTypeString>()};
+ return {std::make_shared<DataTypeString>()};
}
static Status vector(const ColumnString::Chars& data, const ColumnString::Offsets& offsets,
@@ -774,8 +774,8 @@ void register_function_string(SimpleFunctionFactory& factory) {
factory.register_function<FunctionLTrim>();
factory.register_function<FunctionRTrim>();
factory.register_function<FunctionTrim>();
- factory.register_function<FunctionSubstring<Substr3Imp>>();
- factory.register_function<FunctionSubstring<Substr2Imp>>();
+ factory.register_function<FunctionSubstring<Substr3Impl>>();
+ factory.register_function<FunctionSubstring<Substr2Impl>>();
factory.register_function<FunctionLeft>();
factory.register_function<FunctionRight>();
factory.register_function<FunctionNullOrEmpty>();
@@ -792,12 +792,18 @@ void register_function_string(SimpleFunctionFactory& factory) {
factory.register_function<FunctionSplitPart>();
factory.register_function<FunctionStringMd5sum>();
factory.register_function<FunctionStringParseUrl>();
+ factory.register_function<FunctionMoneyFormat<MoneyFormatDoubleImpl>>();
+ factory.register_function<FunctionMoneyFormat<MoneyFormatInt64Impl>>();
+ factory.register_function<FunctionMoneyFormat<MoneyFormatInt128Impl>>();
+ factory.register_function<FunctionMoneyFormat<MoneyFormatDecimalImpl>>();
factory.register_alias(FunctionLeft::name, "strleft");
factory.register_alias(FunctionRight::name, "strright");
factory.register_alias(SubstringUtil::name, "substr");
factory.register_alias(FunctionToLower::name, "lcase");
+ factory.register_alias(FunctionToUpper::name, "ucase");
factory.register_alias(FunctionStringMd5sum::name, "md5");
+ factory.register_alias(FunctionStringUTF8Length::name, "character_length");
}
} // namespace doris::vectorized
diff --git a/be/src/vec/functions/function_string.h b/be/src/vec/functions/function_string.h
index 3f3e538..af58062 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -24,9 +24,12 @@
#include <string_view>
#include "exprs/anyval_util.h"
+#include "exprs/math_functions.h"
+#include "exprs/string_functions.h"
#include "runtime/string_value.hpp"
#include "util/md5.h"
#include "util/url_parser.h"
+#include "vec/columns/column_decimal.h"
#include "vec/columns/column_nullable.h"
#include "vec/columns/column_string.h"
#include "vec/columns/columns_number.h"
@@ -211,7 +214,7 @@ public:
}
};
-struct Substr3Imp {
+struct Substr3Impl {
static DataTypes get_variadic_argument_types() {
return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeInt32>(),
std::make_shared<DataTypeInt32>()};
@@ -225,7 +228,7 @@ struct Substr3Imp {
}
};
-struct Substr2Imp {
+struct Substr2Impl {
static DataTypes get_variadic_argument_types() {
return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeInt32>()};
}
@@ -558,7 +561,7 @@ public:
}
return Status::OK();
}
-}; // namespace doris::vectorized
+};
class FunctionStringRepeat : public IFunction {
public:
@@ -1038,4 +1041,105 @@ public:
}
};
+template <typename Impl>
+class FunctionMoneyFormat : public IFunction {
+public:
+ static constexpr auto name = "money_format";
+ static FunctionPtr create() { return std::make_shared<FunctionMoneyFormat<Impl>>(); }
+ String get_name() const override { return name; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return std::make_shared<DataTypeString>();
+ }
+ DataTypes get_variadic_argument_types_impl() const override {
+ return Impl::get_variadic_argument_types();
+ }
+ size_t get_number_of_arguments() const override { return 1; }
+
+ bool use_default_implementation_for_constants() const override { return true; }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ auto res_column = ColumnString::create();
+ ColumnPtr argument_column = block.get_by_position(arguments[0]).column;
+
+ auto result_column = assert_cast<ColumnString*>(res_column.get());
+ auto data_column = assert_cast<const typename Impl::ColumnType*>(argument_column.get());
+
+ Impl::execute(context, result_column, data_column, input_rows_count);
+
+ block.replace_by_position(result, std::move(res_column));
+ return Status::OK();
+ }
+};
+
+struct MoneyFormatDoubleImpl {
+ using ColumnType = ColumnVector<Float64>;
+
+ static DataTypes get_variadic_argument_types() { return {std::make_shared<DataTypeFloat64>()}; }
+
+ static void execute(FunctionContext* context, ColumnString* result_column,
+ const ColumnType* data_column, size_t input_rows_count) {
+ for (size_t i = 0; i < input_rows_count; i++) {
+ double value =
+ MathFunctions::my_double_round(data_column->get_element(i), 2, false, false);
+ StringVal str = StringFunctions::do_money_format(context, fmt::format("{:.2f}", value));
+ result_column->insert_data(reinterpret_cast<const char*>(str.ptr), str.len);
+ }
+ }
+};
+
+struct MoneyFormatInt64Impl {
+ using ColumnType = ColumnVector<Int64>;
+
+ static DataTypes get_variadic_argument_types() { return {std::make_shared<DataTypeInt64>()}; }
+
+ static void execute(FunctionContext* context, ColumnString* result_column,
+ const ColumnType* data_column, size_t input_rows_count) {
+ for (size_t i = 0; i < input_rows_count; i++) {
+ Int64 value = data_column->get_element(i);
+ StringVal str = StringFunctions::do_money_format<Int64, 26>(context, value);
+ result_column->insert_data(reinterpret_cast<const char*>(str.ptr), str.len);
+ }
+ }
+};
+
+struct MoneyFormatInt128Impl {
+ using ColumnType = ColumnVector<Int128>;
+
+ static DataTypes get_variadic_argument_types() { return {std::make_shared<DataTypeInt128>()}; }
+
+ static void execute(FunctionContext* context, ColumnString* result_column,
+ const ColumnType* data_column, size_t input_rows_count) {
+ for (size_t i = 0; i < input_rows_count; i++) {
+ Int128 value = data_column->get_element(i);
+ StringVal str = StringFunctions::do_money_format<Int128, 52>(context, value);
+ result_column->insert_data(reinterpret_cast<const char*>(str.ptr), str.len);
+ }
+ }
+};
+
+struct MoneyFormatDecimalImpl {
+ using ColumnType = ColumnDecimal<Decimal128>;
+
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeDecimal<Decimal128>>(27, 9)};
+ }
+
+ static void execute(FunctionContext* context, ColumnString* result_column,
+ const ColumnType* data_column, size_t input_rows_count) {
+ for (size_t i = 0; i < input_rows_count; i++) {
+ DecimalV2Val value = DecimalV2Val(data_column->get_element(i));
+
+ DecimalV2Value rounded(0);
+ DecimalV2Value::from_decimal_val(value).round(&rounded, 2, HALF_UP);
+
+ StringVal str = StringFunctions::do_money_format<int64_t, 26>(
+ context, rounded.int_value(), abs(rounded.frac_value() / 10000000));
+
+ result_column->insert_data(reinterpret_cast<const char*>(str.ptr), str.len);
+ }
+ }
+};
+
} // namespace doris::vectorized
\ No newline at end of file
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 5d96751..2fe936a 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -933,16 +933,16 @@ visible_functions = [
'15FunctionContextENS2_18FunctionStateScopeE', 'vec', 'ALWAYS_NULLABLE'],
[['money_format'], 'VARCHAR', ['BIGINT'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_9BigIntValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['money_format'], 'VARCHAR', ['LARGEINT'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_11LargeIntValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['money_format'], 'VARCHAR', ['DOUBLE'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_9DoubleValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['money_format'], 'VARCHAR', ['DECIMALV2'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_12DecimalV2ValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['split_part'], 'VARCHAR', ['VARCHAR', 'VARCHAR', 'INT'],
'_ZN5doris15StringFunctions10split_partEPN9doris_udf15FunctionContextERKNS1_9StringValES6_RKNS1_6IntValE',
'', '', 'vec', 'ALWAYS_NULLABLE'],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 05/33: [Bug] Change parser string to int (#7595)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 45286430724bfecaf73407f15c427aae6bd417e3
Author: Pxl <95...@qq.com>
AuthorDate: Wed Jan 5 16:13:38 2022 +0800
[Bug] Change parser string to int (#7595)
---
be/src/util/string_parser.hpp | 20 +++++++++++++++++
be/src/vec/io/io_helper.h | 52 ++++++-------------------------------------
2 files changed, 27 insertions(+), 45 deletions(-)
diff --git a/be/src/util/string_parser.hpp b/be/src/util/string_parser.hpp
index 0354343..cc1110c 100644
--- a/be/src/util/string_parser.hpp
+++ b/be/src/util/string_parser.hpp
@@ -573,6 +573,26 @@ T StringParser::numeric_limits(bool negative) {
}
template<>
+inline int StringParser::StringParseTraits<uint8_t>::max_ascii_len() {
+ return 3;
+}
+
+template<>
+inline int StringParser::StringParseTraits<uint16_t>::max_ascii_len() {
+ return 5;
+}
+
+template<>
+inline int StringParser::StringParseTraits<uint32_t>::max_ascii_len() {
+ return 10;
+}
+
+template<>
+inline int StringParser::StringParseTraits<uint64_t>::max_ascii_len() {
+ return 20;
+}
+
+template<>
inline int StringParser::StringParseTraits<int8_t>::max_ascii_len() {
return 3;
}
diff --git a/be/src/vec/io/io_helper.h b/be/src/vec/io/io_helper.h
index fc232d7..fb9371f 100644
--- a/be/src/vec/io/io_helper.h
+++ b/be/src/vec/io/io_helper.h
@@ -126,7 +126,7 @@ inline void write_string_binary(const StringRef& s, BufferWritable& buf) {
}
inline void write_string_binary(const char* s, BufferWritable& buf) {
- write_string_binary(StringRef{s}, buf);
+ write_string_binary(StringRef {s}, buf);
}
template <typename Type>
@@ -288,53 +288,15 @@ bool read_float_text_fast_impl(T& x, ReadBuffer& in) {
template <typename T>
bool read_int_text_impl(T& x, ReadBuffer& buf) {
- bool negative = false;
- std::make_unsigned_t<T> res = 0;
- if (buf.eof()) {
- return false;
- }
+ StringParser::ParseResult result;
+ x = StringParser::string_to_int<T>(buf.position(), buf.count(), &result);
- while (!buf.eof()) {
- switch (*buf.position()) {
- case '+':
- break;
- case '-':
- if (std::is_signed_v<T>)
- negative = true;
- else {
- return false;
- }
- break;
- case '0':
- [[fallthrough]];
- case '1':
- [[fallthrough]];
- case '2':
- [[fallthrough]];
- case '3':
- [[fallthrough]];
- case '4':
- [[fallthrough]];
- case '5':
- [[fallthrough]];
- case '6':
- [[fallthrough]];
- case '7':
- [[fallthrough]];
- case '8':
- [[fallthrough]];
- case '9':
- res *= 10;
- res += *buf.position() - '0';
- break;
- default:
- x = negative ? -res : res;
- return true;
- }
- ++buf.position();
+ if (UNLIKELY(result != StringParser::PARSE_SUCCESS)) {
+ return false;
}
- x = negative ? -res : res;
+ // only to match the is_all_read() check to prevent return null
+ buf.position() = buf.end();
return true;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 14/33: [vectorized] [block] Add new method get_data_type to avoid unnecessary copy by the method get_data_type (#7600)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit e4619d98a18952a24e2e3583d3ab82da2d0a1ba8
Author: thinker <zc...@qq.com>
AuthorDate: Fri Jan 7 15:37:35 2022 +0800
[vectorized] [block] Add new method get_data_type to avoid unnecessary copy by the method get_data_type (#7600)
Co-authored-by: zuochunwei <zu...@meituan.com>
---
be/src/vec/core/block.cpp | 4 ++--
be/src/vec/core/block.h | 7 ++++++-
be/src/vec/olap/block_reader.cpp | 4 ++--
3 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/be/src/vec/core/block.cpp b/be/src/vec/core/block.cpp
index a5e445d..2aff751 100644
--- a/be/src/vec/core/block.cpp
+++ b/be/src/vec/core/block.cpp
@@ -51,7 +51,7 @@
namespace doris::vectorized {
-inline DataTypePtr get_data_type(const PColumn& pcolumn) {
+inline DataTypePtr create_data_type(const PColumn& pcolumn) {
switch (pcolumn.type()) {
case PColumn::UINT8: {
return std::make_shared<DataTypeUInt8>();
@@ -176,7 +176,7 @@ Block::Block(const ColumnsWithTypeAndName& data_) : data {data_} {
Block::Block(const PBlock& pblock) {
for (const auto& pcolumn : pblock.columns()) {
- DataTypePtr type = get_data_type(pcolumn);
+ DataTypePtr type = create_data_type(pcolumn);
MutableColumnPtr data_column;
if (pcolumn.is_null_size() > 0) {
data_column =
diff --git a/be/src/vec/core/block.h b/be/src/vec/core/block.h
index addecf7..1c435ee 100644
--- a/be/src/vec/core/block.h
+++ b/be/src/vec/core/block.h
@@ -130,6 +130,11 @@ public:
Names get_names() const;
DataTypes get_data_types() const;
+ DataTypePtr get_data_type(size_t index) const {
+ CHECK(index < data.size());
+ return data[index].type;
+ }
+
/// Returns number of rows from first column in block, not equal to nullptr. If no columns, returns 0.
size_t rows() const;
@@ -204,7 +209,7 @@ public:
static Status filter_block(Block* block, int filter_conlumn_id, int column_to_keep);
static inline void erase_useless_column(Block* block, int column_to_keep) {
- for (size_t i = block->columns() - 1; i >= column_to_keep; --i) {
+ for (int i = block->columns() - 1; i >= column_to_keep; --i) {
block->erase(i);
}
}
diff --git a/be/src/vec/olap/block_reader.cpp b/be/src/vec/olap/block_reader.cpp
index 769e27e..8e2d4b2 100644
--- a/be/src/vec/olap/block_reader.cpp
+++ b/be/src/vec/olap/block_reader.cpp
@@ -94,11 +94,11 @@ void BlockReader::_init_agg_state() {
// create aggregate function
DataTypes argument_types;
- argument_types.push_back(_next_row.block->get_data_types()[idx]);
+ argument_types.push_back(_next_row.block->get_data_type(idx));
Array params;
AggregateFunctionPtr function = AggregateFunctionSimpleFactory::instance().get(
agg_name, argument_types, params,
- _next_row.block->get_data_types()[idx]->is_nullable());
+ _next_row.block->get_data_type(idx)->is_nullable());
DCHECK(function != nullptr);
_agg_functions.push_back(function);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 18/33: [Vectorized] (olap) Optimize BlockReader's performance (#7642)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8b8210433eeeb0fbfc20b39520188d2e23892767
Author: thinker <zc...@qq.com>
AuthorDate: Mon Jan 10 20:28:21 2022 +0800
[Vectorized] (olap) Optimize BlockReader's performance (#7642)
Co-authored-by: zuochunwei <zu...@meituan.com>
---
be/src/vec/olap/block_reader.cpp | 51 +++++++++++++++++-----------------------
be/src/vec/olap/block_reader.h | 9 +++----
2 files changed, 24 insertions(+), 36 deletions(-)
diff --git a/be/src/vec/olap/block_reader.cpp b/be/src/vec/olap/block_reader.cpp
index 8e2d4b2..ef3ba3a 100644
--- a/be/src/vec/olap/block_reader.cpp
+++ b/be/src/vec/olap/block_reader.cpp
@@ -25,15 +25,8 @@
#include "runtime/mem_tracker.h"
#include "vec/olap/vcollect_iterator.h"
-using std::nothrow;
-using std::set;
-using std::vector;
-
namespace doris::vectorized {
-BlockReader::BlockReader()
- : _collect_iter(new VCollectIterator()), _next_row {nullptr, -1, false} {}
-
BlockReader::~BlockReader() {
for (int i = 0; i < _agg_functions.size(); ++i) {
AggregateFunctionPtr function = _agg_functions[i];
@@ -45,7 +38,7 @@ BlockReader::~BlockReader() {
OLAPStatus BlockReader::_init_collect_iter(const ReaderParams& read_params,
std::vector<RowsetReaderSharedPtr>* valid_rs_readers) {
- _collect_iter->init(this);
+ _vcollect_iter.init(this);
std::vector<RowsetReaderSharedPtr> rs_readers;
auto res = _capture_rs_readers(read_params, &rs_readers);
if (res != OLAP_SUCCESS) {
@@ -59,7 +52,7 @@ OLAPStatus BlockReader::_init_collect_iter(const ReaderParams& read_params,
for (auto& rs_reader : rs_readers) {
RETURN_NOT_OK(rs_reader->init(&_reader_context));
- OLAPStatus res = _collect_iter->add_child(rs_reader);
+ OLAPStatus res = _vcollect_iter.add_child(rs_reader);
if (res != OLAP_SUCCESS && res != OLAP_ERR_DATA_EOF) {
LOG(WARNING) << "failed to add child to iterator, err=" << res;
return res;
@@ -69,9 +62,9 @@ OLAPStatus BlockReader::_init_collect_iter(const ReaderParams& read_params,
}
}
- _collect_iter->build_heap(*valid_rs_readers);
- if (_collect_iter->is_merge()) {
- auto status = _collect_iter->current_row(&_next_row);
+ _vcollect_iter.build_heap(*valid_rs_readers);
+ if (_vcollect_iter.is_merge()) {
+ auto status = _vcollect_iter.current_row(&_next_row);
_eof = status == OLAP_ERR_DATA_EOF;
}
@@ -85,8 +78,9 @@ void BlockReader::_init_agg_state() {
_stored_has_null_tag.resize(_stored_data_columns.size());
_stored_has_string_tag.resize(_stored_data_columns.size());
+ auto& tablet_schema = tablet()->tablet_schema();
for (auto idx : _agg_columns_idx) {
- FieldAggregationMethod agg_method = tablet()->tablet_schema().column(idx).aggregation();
+ FieldAggregationMethod agg_method = tablet_schema.column(idx).aggregation();
std::string agg_name =
TabletColumn::get_string_by_aggregation_type(agg_method) + agg_reader_suffix;
std::transform(agg_name.begin(), agg_name.end(), agg_name.begin(),
@@ -159,6 +153,7 @@ OLAPStatus BlockReader::init(const ReaderParams& read_params) {
break;
case KeysType::AGG_KEYS:
_next_block_func = &BlockReader::_agg_key_next_block;
+ _init_agg_state();
break;
default:
DCHECK(false) << "No next row function for type:" << tablet()->keys_type();
@@ -170,7 +165,7 @@ OLAPStatus BlockReader::init(const ReaderParams& read_params) {
OLAPStatus BlockReader::_direct_next_block(Block* block, MemPool* mem_pool, ObjectPool* agg_pool,
bool* eof) {
- auto res = _collect_iter->next(block);
+ auto res = _vcollect_iter.next(block);
if (UNLIKELY(res != OLAP_SUCCESS && res != OLAP_ERR_DATA_EOF)) {
return res;
}
@@ -190,11 +185,6 @@ OLAPStatus BlockReader::_agg_key_next_block(Block* block, MemPool* mem_pool, Obj
return OLAP_SUCCESS;
}
- if (!_agg_inited) {
- _init_agg_state();
- _agg_inited = true;
- }
-
auto target_block_row = 0;
auto target_columns = block->mutate_columns();
@@ -203,7 +193,7 @@ OLAPStatus BlockReader::_agg_key_next_block(Block* block, MemPool* mem_pool, Obj
_append_agg_data(target_columns);
while (true) {
- auto res = _collect_iter->next(&_next_row);
+ auto res = _vcollect_iter.next(&_next_row);
if (UNLIKELY(res == OLAP_ERR_DATA_EOF)) {
*eof = true;
break;
@@ -251,7 +241,7 @@ OLAPStatus BlockReader::_unique_key_next_block(Block* block, MemPool* mem_pool,
// the version is in reverse order, the first row is the highest version,
// in UNIQUE_KEY highest version is the final result, there is no need to
// merge the lower versions
- auto res = _collect_iter->next(&_next_row);
+ auto res = _vcollect_iter.next(&_next_row);
if (UNLIKELY(res == OLAP_ERR_DATA_EOF)) {
*eof = true;
break;
@@ -268,9 +258,9 @@ OLAPStatus BlockReader::_unique_key_next_block(Block* block, MemPool* mem_pool,
}
void BlockReader::_insert_data_normal(MutableColumns& columns) {
+ auto block = _next_row.block;
for (auto idx : _normal_columns_idx) {
- columns[_return_columns_loc[idx]]->insert_from(
- *_next_row.block->get_by_position(idx).column, _next_row.row_pos);
+ columns[_return_columns_loc[idx]]->insert_from(*block->get_by_position(idx).column, _next_row.row_pos);
}
}
@@ -279,7 +269,7 @@ void BlockReader::_append_agg_data(MutableColumns& columns) {
_last_agg_data_counter++;
// execute aggregate when have `batch_size` column or some ref invalid soon
- bool is_last = (_stored_row_ref.back().block->rows() == _stored_row_ref.back().row_pos + 1);
+ bool is_last = (_next_row.block->rows() == _next_row.row_pos + 1);
if (_stored_row_ref.size() == _batch_size || is_last) {
_update_agg_data(columns);
}
@@ -314,23 +304,24 @@ void BlockReader::_copy_agg_data() {
phmap::flat_hash_map<const Block*, std::vector<std::pair<int16_t, int16_t>>> temp_ref_map;
for (int i = 0; i < _stored_row_ref.size(); i++) {
- auto ref = _stored_row_ref[i];
- temp_ref_map[ref.block].push_back({ref.row_pos, i});
+ auto& ref = _stored_row_ref[i];
+ temp_ref_map[ref.block].emplace_back(ref.row_pos, i);
}
for (auto idx : _agg_columns_idx) {
+ auto& dst_column = _stored_data_columns[idx];
if (_stored_has_string_tag[idx]) {
//string type should replace ordered
for (int i = 0; i < _stored_row_ref.size(); i++) {
- auto ref = _stored_row_ref[i];
- _stored_data_columns[idx]->replace_column_data(
+ auto& ref = _stored_row_ref[i];
+ dst_column->replace_column_data(
*ref.block->get_by_position(idx).column, ref.row_pos, i);
}
} else {
for (auto& it : temp_ref_map) {
+ auto& src_column = *it.first->get_by_position(idx).column;
for (auto& pos : it.second) {
- _stored_data_columns[idx]->replace_column_data(
- *it.first->get_by_position(idx).column, pos.first, pos.second);
+ dst_column->replace_column_data(src_column, pos.first, pos.second);
}
}
}
diff --git a/be/src/vec/olap/block_reader.h b/be/src/vec/olap/block_reader.h
index c8c566f..9199072 100644
--- a/be/src/vec/olap/block_reader.h
+++ b/be/src/vec/olap/block_reader.h
@@ -34,8 +34,6 @@ namespace vectorized {
class BlockReader final : public Reader {
public:
- BlockReader();
-
~BlockReader();
// Initialize BlockReader with tablet, data version and fetch range.
@@ -87,8 +85,8 @@ private:
void _update_agg_value(MutableColumns& columns, int begin, int end, bool is_close = true);
- std::unique_ptr<VCollectIterator> _collect_iter;
- IteratorRowRef _next_row;
+ VCollectIterator _vcollect_iter;
+ IteratorRowRef _next_row{nullptr, -1, false};
std::vector<AggregateFunctionPtr> _agg_functions;
std::vector<AggregateDataPtr> _agg_places;
@@ -97,7 +95,7 @@ private:
std::vector<int> _agg_columns_idx;
std::vector<int> _return_columns_loc;
- int _batch_size;
+ int _batch_size = 0;
std::vector<int> _agg_data_counters;
int _last_agg_data_counter = 0;
@@ -110,7 +108,6 @@ private:
std::vector<bool> _stored_has_string_tag;
bool _eof = false;
- bool _agg_inited = false;
OLAPStatus (BlockReader::*_next_block_func)(Block* block, MemPool* mem_pool,
ObjectPool* agg_pool, bool* eof) = nullptr;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 20/33: [Vectorized][Enhancement] fix some bug & improve some code (#7714)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit b20b5b7e4310a91c6cb42a8cc1ba4c6850bd3af2
Author: Pxl <95...@qq.com>
AuthorDate: Wed Jan 12 09:57:10 2022 +0800
[Vectorized][Enhancement] fix some bug & improve some code (#7714)
---
.../aggregate_function_reader.cpp | 8 +++-----
be/src/vec/exec/volap_scan_node.cpp | 2 ++
be/src/vec/olap/block_reader.cpp | 22 ++++++++++++++--------
be/src/vec/olap/block_reader.h | 4 +++-
run-be-ut.sh | 2 +-
5 files changed, 23 insertions(+), 15 deletions(-)
diff --git a/be/src/vec/aggregate_functions/aggregate_function_reader.cpp b/be/src/vec/aggregate_functions/aggregate_function_reader.cpp
index 9a24ac5..3594d51 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_reader.cpp
+++ b/be/src/vec/aggregate_functions/aggregate_function_reader.cpp
@@ -23,9 +23,8 @@ namespace doris::vectorized {
void register_aggregate_function_reader(AggregateFunctionSimpleFactory& factory) {
// add a suffix to the function name here to distinguish special functions of agg reader
auto register_function_reader = [&](const std::string& name,
- const AggregateFunctionCreator& creator,
- bool nullable = false) {
- factory.register_function(name + agg_reader_suffix, creator, nullable);
+ const AggregateFunctionCreator& creator) {
+ factory.register_function(name + agg_reader_suffix, creator, false);
};
register_function_reader("sum", create_aggregate_function_sum_reader);
@@ -38,8 +37,7 @@ void register_aggregate_function_reader(AggregateFunctionSimpleFactory& factory)
void register_aggregate_function_reader_no_spread(AggregateFunctionSimpleFactory& factory) {
auto register_function_reader = [&](const std::string& name,
- const AggregateFunctionCreator& creator,
- bool nullable = false) {
+ const AggregateFunctionCreator& creator, bool nullable) {
factory.register_function(name + agg_reader_suffix, creator, nullable);
};
diff --git a/be/src/vec/exec/volap_scan_node.cpp b/be/src/vec/exec/volap_scan_node.cpp
index da7a204..b365c1d 100644
--- a/be/src/vec/exec/volap_scan_node.cpp
+++ b/be/src/vec/exec/volap_scan_node.cpp
@@ -259,6 +259,8 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
}
_scan_cpu_timer->update(cpu_watch.elapsed_time());
_scanner_wait_worker_timer->update(wait_time);
+
+ std::unique_lock<std::mutex> l(_scan_blocks_lock);
_running_thread--;
// The transfer thead will wait for `_running_thread==0`, to make sure all scanner threads won't access class members.
diff --git a/be/src/vec/olap/block_reader.cpp b/be/src/vec/olap/block_reader.cpp
index ef3ba3a..f7a1388 100644
--- a/be/src/vec/olap/block_reader.cpp
+++ b/be/src/vec/olap/block_reader.cpp
@@ -72,6 +72,10 @@ OLAPStatus BlockReader::_init_collect_iter(const ReaderParams& read_params,
}
void BlockReader::_init_agg_state() {
+ if (_eof) {
+ return;
+ }
+
_stored_data_block = _next_row.block->create_same_struct_block(_batch_size);
_stored_data_columns = _stored_data_block->mutate_columns();
@@ -260,7 +264,8 @@ OLAPStatus BlockReader::_unique_key_next_block(Block* block, MemPool* mem_pool,
void BlockReader::_insert_data_normal(MutableColumns& columns) {
auto block = _next_row.block;
for (auto idx : _normal_columns_idx) {
- columns[_return_columns_loc[idx]]->insert_from(*block->get_by_position(idx).column, _next_row.row_pos);
+ columns[_return_columns_loc[idx]]->insert_from(*block->get_by_position(idx).column,
+ _next_row.row_pos);
}
}
@@ -270,7 +275,7 @@ void BlockReader::_append_agg_data(MutableColumns& columns) {
// execute aggregate when have `batch_size` column or some ref invalid soon
bool is_last = (_next_row.block->rows() == _next_row.row_pos + 1);
- if (_stored_row_ref.size() == _batch_size || is_last) {
+ if (is_last || _stored_row_ref.size() == _batch_size) {
_update_agg_data(columns);
}
}
@@ -301,11 +306,9 @@ void BlockReader::_update_agg_data(MutableColumns& columns) {
}
void BlockReader::_copy_agg_data() {
- phmap::flat_hash_map<const Block*, std::vector<std::pair<int16_t, int16_t>>> temp_ref_map;
-
for (int i = 0; i < _stored_row_ref.size(); i++) {
auto& ref = _stored_row_ref[i];
- temp_ref_map[ref.block].emplace_back(ref.row_pos, i);
+ _temp_ref_map[ref.block].emplace_back(ref.row_pos, i);
}
for (auto idx : _agg_columns_idx) {
@@ -314,11 +317,11 @@ void BlockReader::_copy_agg_data() {
//string type should replace ordered
for (int i = 0; i < _stored_row_ref.size(); i++) {
auto& ref = _stored_row_ref[i];
- dst_column->replace_column_data(
- *ref.block->get_by_position(idx).column, ref.row_pos, i);
+ dst_column->replace_column_data(*ref.block->get_by_position(idx).column,
+ ref.row_pos, i);
}
} else {
- for (auto& it : temp_ref_map) {
+ for (auto& it : _temp_ref_map) {
auto& src_column = *it.first->get_by_position(idx).column;
for (auto& pos : it.second) {
dst_column->replace_column_data(src_column, pos.first, pos.second);
@@ -327,6 +330,9 @@ void BlockReader::_copy_agg_data() {
}
}
+ for (auto& it : _temp_ref_map) {
+ it.second.clear();
+ }
_stored_row_ref.clear();
}
diff --git a/be/src/vec/olap/block_reader.h b/be/src/vec/olap/block_reader.h
index 9199072..02cb570 100644
--- a/be/src/vec/olap/block_reader.h
+++ b/be/src/vec/olap/block_reader.h
@@ -86,7 +86,7 @@ private:
void _update_agg_value(MutableColumns& columns, int begin, int end, bool is_close = true);
VCollectIterator _vcollect_iter;
- IteratorRowRef _next_row{nullptr, -1, false};
+ IteratorRowRef _next_row {nullptr, -1, false};
std::vector<AggregateFunctionPtr> _agg_functions;
std::vector<AggregateDataPtr> _agg_places;
@@ -107,6 +107,8 @@ private:
std::vector<bool> _stored_has_null_tag;
std::vector<bool> _stored_has_string_tag;
+ phmap::flat_hash_map<const Block*, std::vector<std::pair<int16_t, int16_t>>> _temp_ref_map;
+
bool _eof = false;
OLAPStatus (BlockReader::*_next_block_func)(Block* block, MemPool* mem_pool,
diff --git a/run-be-ut.sh b/run-be-ut.sh
index 1fce850..7af98f7 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -73,7 +73,7 @@ fi
eval set -- "$OPTS"
-PARALLEL=$[$(nproc)/4+1]
+PARALLEL=$[$(nproc)/8+1]
CLEAN=
RUN=
if [ $# == 1 ] ; then
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 01/33: [Vectorized Exec Engine] Support Vectorized Exec Engine In Doris
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 14d84ba0c9b43e2daa91f8fad4b7f37ed3b90c97
Author: lihaopeng <li...@baidu.com>
AuthorDate: Fri Dec 31 11:31:31 2021 +0800
[Vectorized Exec Engine] Support Vectorized Exec Engine In Doris
Co-authored-by: HappenLee <ha...@hotmail.com>
Co-authored-by: stdpain <34...@users.noreply.github.com>
Co-authored-by: Zhengguo Yang <ya...@gmail.com>
Co-authored-by: wangbo <50...@qq.com>
Co-authored-by: emmymiao87 <52...@qq.com>
Co-authored-by: Pxl <95...@qq.com>
Co-authored-by: zhangstar333 <87...@users.noreply.github.com>
Co-authored-by: thinker <zc...@qq.com>
Co-authored-by: Zeno Yang <15...@qq.com>
Co-authored-by: Wang Shuo <wa...@gmail.com>
Co-authored-by: zhoubintao <35...@users.noreply.github.com>
Co-authored-by: Gabriel <ga...@gmail.com>
Co-authored-by: xinghuayu007 <14...@qq.com>
Co-authored-by: weizuo93 <we...@apache.org>
Co-authored-by: yiguolei <gu...@tencent.com>
Co-authored-by: anneji-dev <85...@users.noreply.github.com>
Co-authored-by: awakeljw <99...@qq.com>
Co-authored-by: taberylyang <95...@users.noreply.github.com>
Co-authored-by: Cui Kaifeng <48...@users.noreply.github.com>
---
be/CMakeLists.txt | 12 +
be/src/exec/blocking_join_node.cpp | 3 +-
be/src/exec/data_sink.cpp | 15 +-
be/src/exec/data_sink.h | 11 +-
be/src/exec/es/es_scroll_parser.cpp | 381 +++++-
be/src/exec/es/es_scroll_parser.h | 8 +
be/src/exec/es_http_scan_node.cpp | 15 +-
be/src/exec/es_http_scan_node.h | 45 +-
be/src/exec/es_http_scanner.h | 2 +-
be/src/exec/exec_node.cpp | 152 ++-
be/src/exec/exec_node.h | 13 +
be/src/exec/mysql_scan_node.h | 2 +-
be/src/exec/odbc_scan_node.cpp | 12 +-
be/src/exec/odbc_scan_node.h | 12 +-
be/src/exec/olap_scan_node.cpp | 57 +-
be/src/exec/olap_scan_node.h | 8 +-
be/src/exec/olap_scanner.cpp | 7 +-
be/src/exec/partitioned_aggregation_node.cc | 2 +-
be/src/exec/scan_node.h | 2 +-
be/src/exec/schema_scan_node.h | 2 +-
be/src/exec/tablet_info.cpp | 224 ++++
be/src/exec/tablet_info.h | 100 ++
be/src/exec/tablet_sink.cpp | 72 +-
be/src/exec/tablet_sink.h | 6 +-
be/src/exec/text_converter.cpp | 22 +
be/src/exec/text_converter.h | 7 +-
be/src/exec/text_converter.hpp | 133 +-
be/src/exprs/CMakeLists.txt | 1 -
be/src/exprs/aggregate_functions.cpp | 20 +-
be/src/exprs/math_functions.cpp | 6 +-
be/src/exprs/math_functions.h | 2 +-
be/src/exprs/runtime_filter.cpp | 34 +
be/src/exprs/runtime_filter.h | 1 +
be/src/exprs/runtime_filter_slots.h | 38 +-
be/src/exprs/v_string_functions.h | 219 ++++
be/src/olap/block_column_predicate.cpp | 114 ++
be/src/olap/block_column_predicate.h | 26 +
be/src/olap/collect_iterator.cpp | 2 +-
be/src/olap/column_predicate.h | 13 +
be/src/olap/comparison_predicate.cpp | 163 +++
be/src/olap/comparison_predicate.h | 4 +
be/src/olap/generic_iterators.cpp | 62 +-
be/src/olap/generic_iterators.h | 4 +-
be/src/olap/iterators.h | 6 +-
be/src/olap/null_predicate.cpp | 43 +
be/src/olap/null_predicate.h | 8 +
be/src/olap/reader.cpp | 11 +-
be/src/olap/reader.h | 24 +-
be/src/olap/row_block2.cpp | 208 ++++
be/src/olap/row_block2.h | 5 +
be/src/olap/rowset/alpha_rowset_reader.h | 4 +
be/src/olap/rowset/beta_rowset_reader.cpp | 42 +-
be/src/olap/rowset/beta_rowset_reader.h | 1 +
be/src/olap/rowset/rowset_reader.h | 8 +
be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 52 +-
be/src/olap/rowset/segment_v2/binary_dict_page.h | 9 +
be/src/olap/rowset/segment_v2/binary_plain_page.h | 31 +
be/src/olap/rowset/segment_v2/binary_prefix_page.h | 4 +
be/src/olap/rowset/segment_v2/bitshuffle_page.h | 48 +
be/src/olap/rowset/segment_v2/column_reader.cpp | 103 ++
be/src/olap/rowset/segment_v2/column_reader.h | 20 +
.../rowset/segment_v2/empty_segment_iterator.cpp | 6 +-
.../rowset/segment_v2/empty_segment_iterator.h | 3 +-
.../rowset/segment_v2/frame_of_reference_page.h | 4 +
be/src/olap/rowset/segment_v2/page_decoder.h | 5 +
be/src/olap/rowset/segment_v2/plain_page.h | 4 +
be/src/olap/rowset/segment_v2/rle_page.h | 23 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 5 +
be/src/olap/rowset/segment_v2/segment_iterator.h | 3 +
be/src/olap/schema.cpp | 44 +
be/src/olap/schema.h | 4 +
be/src/olap/tablet_schema.cpp | 13 +
be/src/olap/tablet_schema.h | 4 +
be/src/olap/tuple_reader.cpp | 2 +-
be/src/runtime/datetime_value.cpp | 2 +
be/src/runtime/datetime_value.h | 6 +-
be/src/runtime/descriptors.cpp | 21 +-
be/src/runtime/descriptors.h | 5 +-
be/src/runtime/exec_env.h | 2 +
be/src/runtime/exec_env_init.cpp | 2 +
be/src/runtime/fold_constant_executor.cpp | 82 +-
be/src/runtime/fold_constant_executor.h | 7 +-
be/src/runtime/mysql_result_writer.cpp | 8 +-
be/src/runtime/mysql_result_writer.h | 2 +
be/src/runtime/plan_fragment_executor.cpp | 93 +-
be/src/runtime/plan_fragment_executor.h | 5 +-
be/src/runtime/primitive_type.h | 77 +-
be/src/runtime/raw_value.h | 56 +
be/src/runtime/result_sink.cpp | 2 +
be/src/runtime/row_batch.cpp | 56 +-
be/src/runtime/row_batch.h | 5 +-
be/src/runtime/string_value.hpp | 25 +-
be/src/runtime/types.h | 58 +
be/src/service/internal_service.cpp | 7 +-
be/src/udf/udf.cpp | 5 +
be/src/udf/udf.h | 5 +
be/src/udf/udf_internal.h | 5 +
be/src/udf/udf_ir.cpp | 14 +
be/src/util/binary_cast.hpp | 17 +-
be/src/util/bitmap_value.h | 6 +
be/src/util/brpc_stub_cache.h | 2 +-
be/src/util/runtime_profile.h | 5 +-
be/src/util/static_asserts.cpp | 3 +-
be/src/util/url_coding.cpp | 4 +
be/src/vec/CMakeLists.txt | 153 +++
.../vec/aggregate_functions/aggregate_function.h | 237 ++++
.../aggregate_functions/aggregate_function_avg.cpp | 68 +
.../aggregate_functions/aggregate_function_avg.h | 128 ++
.../aggregate_function_bitmap.cpp | 87 ++
.../aggregate_function_bitmap.h | 175 +++
.../aggregate_function_combinator.h | 81 ++
.../aggregate_function_count.cpp | 52 +
.../aggregate_functions/aggregate_function_count.h | 122 ++
.../aggregate_function_distinct.cpp | 96 ++
.../aggregate_function_distinct.h | 224 ++++
.../aggregate_function_hll_union_agg.cpp | 51 +
.../aggregate_function_hll_union_agg.h | 133 ++
.../aggregate_function_min_max.cpp | 85 ++
.../aggregate_function_min_max.h | 535 ++++++++
.../aggregate_function_nothing.h | 71 ++
.../aggregate_function_null.cpp | 92 ++
.../aggregate_functions/aggregate_function_null.h | 291 +++++
.../aggregate_function_reader.cpp | 50 +
.../aggregate_function_reader.h} | 30 +-
.../aggregate_function_simple_factory.cpp | 62 +
.../aggregate_function_simple_factory.h | 104 ++
.../aggregate_functions/aggregate_function_sum.cpp | 88 ++
.../aggregate_functions/aggregate_function_sum.h | 118 ++
.../aggregate_function_uniq.cpp | 69 ++
.../aggregate_functions/aggregate_function_uniq.h | 131 ++
.../aggregate_function_window.cpp | 176 +++
.../aggregate_function_window.h | 434 +++++++
be/src/vec/aggregate_functions/factory_helpers.h | 59 +
be/src/vec/aggregate_functions/helpers.h | 225 ++++
.../vec/aggregate_functions/key_holder_helpers.h | 48 +
be/src/vec/columns/collator.cpp | 97 ++
.../columns/collator.h} | 34 +-
be/src/vec/columns/column.cpp | 57 +
be/src/vec/columns/column.h | 483 ++++++++
be/src/vec/columns/column_complex.h | 287 +++++
be/src/vec/columns/column_const.cpp | 107 ++
be/src/vec/columns/column_const.h | 174 +++
be/src/vec/columns/column_decimal.cpp | 224 ++++
be/src/vec/columns/column_decimal.h | 210 ++++
be/src/vec/columns/column_dummy.h | 141 +++
be/src/vec/columns/column_impl.h | 62 +
.../columns/column_nothing.h} | 37 +-
be/src/vec/columns/column_nullable.cpp | 438 +++++++
be/src/vec/columns/column_nullable.h | 238 ++++
be/src/vec/columns/column_set.h | 52 +
be/src/vec/columns/column_string.cpp | 375 ++++++
be/src/vec/columns/column_string.h | 286 +++++
be/src/vec/columns/column_vector.cpp | 379 ++++++
be/src/vec/columns/column_vector.h | 268 ++++
be/src/vec/columns/column_vector_helper.h | 59 +
be/src/vec/columns/columns_common.cpp | 273 ++++
be/src/vec/columns/columns_common.h | 91 ++
.../columns/columns_number.h} | 40 +-
be/src/vec/columns/predicate_column.h | 418 +++++++
be/src/vec/common/aggregation_common.h | 269 ++++
be/src/vec/common/allocator.h | 312 +++++
.../common/allocator_fwd.h} | 26 +-
be/src/vec/common/arena.h | 285 +++++
be/src/vec/common/arithmetic_overflow.h | 112 ++
be/src/vec/common/assert_cast.h | 56 +
be/src/vec/common/bit_cast.h | 47 +
be/src/vec/common/bit_helpers.h | 92 ++
be/src/vec/common/columns_hashing.h | 242 ++++
be/src/vec/common/columns_hashing_impl.h | 324 +++++
be/src/vec/common/cow.h | 323 +++++
be/src/vec/common/demangle.cpp | 61 +
.../common/demangle.h} | 26 +-
be/src/vec/common/exception.cpp | 202 +++
be/src/vec/common/exception.h | 287 +++++
be/src/vec/common/field_visitors.h | 554 +++++++++
be/src/vec/common/hash_table/fixed_hash_map.h | 194 +++
be/src/vec/common/hash_table/fixed_hash_table.h | 383 ++++++
be/src/vec/common/hash_table/hash.h | 191 +++
be/src/vec/common/hash_table/hash_map.h | 240 ++++
be/src/vec/common/hash_table/hash_set.h | 105 ++
be/src/vec/common/hash_table/hash_table.h | 914 ++++++++++++++
.../common/hash_table/hash_table_allocator.h} | 32 +-
.../vec/common/hash_table/hash_table_key_holder.h | 139 +++
be/src/vec/common/int_exp.h | 135 ++
be/src/vec/common/memcmp_small.h | 228 ++++
be/src/vec/common/memcpy_small.h | 83 ++
be/src/vec/common/mremap.h | 77 ++
be/src/vec/common/nan_utils.h | 69 ++
.../common/pod_array.cpp} | 23 +-
be/src/vec/common/pod_array.h | 591 +++++++++
be/src/vec/common/pod_array_fwd.h | 53 +
be/src/vec/common/radix_sort.h | 389 ++++++
be/src/vec/common/sip_hash.h | 231 ++++
be/src/vec/common/string_buffer.hpp | 93 ++
be/src/vec/common/string_ref.h | 327 +++++
.../common/string_utils/string_utils.cpp} | 24 +-
be/src/vec/common/string_utils/string_utils.h | 155 +++
be/src/vec/common/strong_typedef.h | 84 ++
be/src/vec/common/typeid_cast.h | 62 +
be/src/vec/common/uint128.h | 237 ++++
be/src/vec/common/unaligned.h | 42 +
be/src/vec/core/accurate_comparison.h | 567 +++++++++
be/src/vec/core/block.cpp | 910 ++++++++++++++
be/src/vec/core/block.h | 349 ++++++
.../core/block_info.cpp} | 30 +-
be/src/vec/core/block_info.h | 76 ++
be/src/vec/core/call_on_type_index.h | 260 ++++
.../core/column_numbers.h} | 20 +-
be/src/vec/core/column_with_type_and_name.cpp | 72 ++
be/src/vec/core/column_with_type_and_name.h | 56 +
.../core/columns_with_type_and_name.h} | 21 +-
be/src/vec/core/decimal_comparison.h | 303 +++++
be/src/vec/core/field.cpp | 193 +++
be/src/vec/core/field.h | 909 ++++++++++++++
.../core/materialize_block.cpp} | 31 +-
.../core/materialize_block.h} | 36 +-
.../empty_segment_iterator.h => vec/core/names.h} | 35 +-
be/src/vec/core/sort_block.cpp | 202 +++
be/src/vec/core/sort_block.h | 55 +
be/src/vec/core/sort_cursor.h | 228 ++++
be/src/vec/core/sort_description.h | 80 ++
be/src/vec/core/types.h | 438 +++++++
be/src/vec/data_types/data_type.cpp | 217 ++++
be/src/vec/data_types/data_type.h | 402 ++++++
be/src/vec/data_types/data_type_bitmap.cpp | 93 ++
be/src/vec/data_types/data_type_bitmap.h | 86 ++
be/src/vec/data_types/data_type_date.cpp | 66 +
be/src/vec/data_types/data_type_date.h | 42 +
be/src/vec/data_types/data_type_date_time.cpp | 95 ++
be/src/vec/data_types/data_type_date_time.h | 83 ++
be/src/vec/data_types/data_type_decimal.cpp | 146 +++
be/src/vec/data_types/data_type_decimal.h | 351 ++++++
be/src/vec/data_types/data_type_factory.hpp | 87 ++
.../data_types/data_type_nothing.cpp} | 30 +-
be/src/vec/data_types/data_type_nothing.h | 65 +
be/src/vec/data_types/data_type_nullable.cpp | 113 ++
be/src/vec/data_types/data_type_nullable.h | 90 ++
be/src/vec/data_types/data_type_number.h | 87 ++
be/src/vec/data_types/data_type_number_base.cpp | 123 ++
be/src/vec/data_types/data_type_number_base.h | 66 +
be/src/vec/data_types/data_type_string.cpp | 115 ++
be/src/vec/data_types/data_type_string.h | 57 +
be/src/vec/data_types/get_least_supertype.cpp | 322 +++++
.../data_types/get_least_supertype.h} | 27 +-
be/src/vec/data_types/nested_utils.cpp | 73 ++
be/src/vec/data_types/nested_utils.h | 43 +
be/src/vec/data_types/number_traits.h | 275 ++++
be/src/vec/exec/join/join_op.h | 143 +++
be/src/vec/exec/join/vacquire_list.hpp | 54 +
be/src/vec/exec/join/vhash_join_node.cpp | 1063 ++++++++++++++++
be/src/vec/exec/join/vhash_join_node.h | 225 ++++
be/src/vec/exec/vaggregation_node.cpp | 1102 +++++++++++++++++
be/src/vec/exec/vaggregation_node.h | 483 ++++++++
be/src/vec/exec/vanalytic_eval_node.cpp | 658 ++++++++++
be/src/vec/exec/vanalytic_eval_node.h | 146 +++
be/src/vec/exec/vassert_num_rows_node.cpp | 99 ++
.../exec/vassert_num_rows_node.h} | 32 +-
.../exec/vblocking_join_node.cpp} | 104 +-
be/src/vec/exec/vblocking_join_node.h | 132 ++
be/src/vec/exec/vcross_join_node.cpp | 185 +++
be/src/vec/exec/vcross_join_node.h | 85 ++
.../exec/vempty_set_node.cpp} | 24 +-
.../exec/vempty_set_node.h} | 29 +-
be/src/vec/exec/ves_http_scan_node.cpp | 176 +++
.../exec/ves_http_scan_node.h} | 36 +-
be/src/vec/exec/ves_http_scanner.cpp | 56 +
.../exec/ves_http_scanner.h} | 33 +-
be/src/vec/exec/vexcept_node.cpp | 112 ++
.../exec/vexcept_node.h} | 28 +-
be/src/vec/exec/vexchange_node.cpp | 104 ++
be/src/vec/exec/vexchange_node.h | 58 +
be/src/vec/exec/vintersect_node.cpp | 111 ++
be/src/vec/exec/vintersect_node.h | 49 +
be/src/vec/exec/vmysql_scan_node.cpp | 124 ++
.../exec/vmysql_scan_node.h} | 36 +-
be/src/vec/exec/vodbc_scan_node.cpp | 130 ++
.../exec/vodbc_scan_node.h} | 23 +-
be/src/vec/exec/volap_scan_node.cpp | 557 +++++++++
be/src/vec/exec/volap_scan_node.h | 69 ++
be/src/vec/exec/volap_scanner.cpp | 205 +++
be/src/vec/exec/volap_scanner.h | 51 +
be/src/vec/exec/vschema_scan_node.cpp | 263 ++++
.../exec/vschema_scan_node.h} | 37 +-
be/src/vec/exec/vselect_node.cpp | 72 ++
.../exec/vselect_node.h} | 28 +-
be/src/vec/exec/vset_operation_node.cpp | 379 ++++++
be/src/vec/exec/vset_operation_node.h | 245 ++++
be/src/vec/exec/vsort_exec_exprs.cpp | 79 ++
be/src/vec/exec/vsort_exec_exprs.h | 98 ++
be/src/vec/exec/vsort_node.cpp | 255 ++++
be/src/vec/exec/vsort_node.h | 92 ++
be/src/vec/exec/vunion_node.cpp | 268 ++++
be/src/vec/exec/vunion_node.h | 109 ++
be/src/vec/exprs/vcase_expr.cpp | 114 ++
be/src/vec/exprs/vcase_expr.h | 49 +
be/src/vec/exprs/vcast_expr.cpp | 97 ++
be/src/vec/exprs/vcast_expr.h | 53 +
.../exprs/vcompound_pred.h} | 39 +-
be/src/vec/exprs/vectorized_agg_fn.cpp | 175 +++
be/src/vec/exprs/vectorized_agg_fn.h | 106 ++
be/src/vec/exprs/vectorized_fn_call.cpp | 120 ++
be/src/vec/exprs/vectorized_fn_call.h | 46 +
be/src/vec/exprs/vexpr.cpp | 343 +++++
be/src/vec/exprs/vexpr.h | 168 +++
be/src/vec/exprs/vexpr_context.cpp | 137 ++
be/src/vec/exprs/vexpr_context.h | 91 ++
be/src/vec/exprs/vin_predicate.cpp | 109 ++
be/src/vec/exprs/vin_predicate.h | 52 +
be/src/vec/exprs/vinfo_func.cpp | 59 +
.../exprs/vinfo_func.h} | 34 +-
be/src/vec/exprs/vliteral.cpp | 138 +++
.../exprs/vliteral.h} | 32 +-
be/src/vec/exprs/vslot_ref.cpp | 75 ++
be/src/vec/exprs/vslot_ref.h | 50 +
.../functions/cast_type_to_either.h} | 23 +-
.../functions/comparison.cpp} | 27 +-
be/src/vec/functions/comparison_equal_for_null.cpp | 125 ++
.../functions/comparison_equals.cpp} | 23 +-
.../functions/comparison_greater.cpp} | 23 +-
.../functions/comparison_less.cpp} | 23 +-
be/src/vec/functions/date_time_transforms.h | 259 ++++
be/src/vec/functions/divide.cpp | 48 +
be/src/vec/functions/function.cpp | 330 +++++
be/src/vec/functions/function.h | 591 +++++++++
.../vec/functions/function_always_not_nullable.h | 85 ++
be/src/vec/functions/function_binary_arithmetic.h | 719 +++++++++++
.../function_binary_arithmetic_to_null_type.h | 247 ++++
be/src/vec/functions/function_bit.cpp | 92 ++
be/src/vec/functions/function_bitmap.cpp | 471 +++++++
.../functions/function_case.cpp} | 21 +-
be/src/vec/functions/function_case.h | 371 ++++++
.../functions/function_cast.cpp} | 25 +-
be/src/vec/functions/function_cast.h | 1307 ++++++++++++++++++++
be/src/vec/functions/function_const.h | 101 ++
.../function_date_or_datetime_computation.cpp | 140 +++
.../function_date_or_datetime_computation.h | 488 ++++++++
.../function_date_or_datetime_to_something.h | 93 ++
.../function_date_or_datetime_to_string.cpp} | 23 +-
.../function_date_or_datetime_to_string.h | 66 +
.../function_datetime_string_to_string.cpp} | 20 +-
.../functions/function_datetime_string_to_string.h | 99 ++
be/src/vec/functions/function_hash.cpp | 254 ++++
.../functions/function_hash.h} | 35 +-
be/src/vec/functions/function_helpers.cpp | 99 ++
be/src/vec/functions/function_helpers.h | 104 ++
.../functions/function_ifnull.cpp} | 23 +-
be/src/vec/functions/function_ifnull.h | 98 ++
be/src/vec/functions/function_json.cpp | 369 ++++++
.../vec/functions/function_math_binary_float64.h | 248 ++++
be/src/vec/functions/function_math_unary.h | 167 +++
be/src/vec/functions/function_string.cpp | 802 ++++++++++++
be/src/vec/functions/function_string.h | 986 +++++++++++++++
be/src/vec/functions/function_string_to_string.h | 78 ++
be/src/vec/functions/function_timestamp.cpp | 330 +++++
be/src/vec/functions/function_totype.h | 428 +++++++
be/src/vec/functions/function_unary_arithmetic.h | 157 +++
be/src/vec/functions/function_variadic_arguments.h | 79 ++
be/src/vec/functions/functions_comparison.h | 475 +++++++
be/src/vec/functions/functions_logical.cpp | 531 ++++++++
be/src/vec/functions/functions_logical.h | 167 +++
be/src/vec/functions/hll_cardinality.cpp | 71 ++
.../functions/hll_empty.cpp} | 38 +-
be/src/vec/functions/hll_hash.cpp | 100 ++
be/src/vec/functions/if.cpp | 480 +++++++
be/src/vec/functions/in.cpp | 182 +++
be/src/vec/functions/int_div.cpp | 153 +++
be/src/vec/functions/int_div.h | 56 +
be/src/vec/functions/is_not_null.cpp | 76 ++
be/src/vec/functions/is_null.cpp | 77 ++
be/src/vec/functions/like.cpp | 321 +++++
be/src/vec/functions/like.h | 155 +++
be/src/vec/functions/math.cpp | 481 +++++++
be/src/vec/functions/minus.cpp | 56 +
be/src/vec/functions/modulo.cpp | 159 +++
be/src/vec/functions/multiply.cpp | 56 +
be/src/vec/functions/nullif.cpp | 155 +++
be/src/vec/functions/plus.cpp | 57 +
be/src/vec/functions/random.cpp | 109 ++
be/src/vec/functions/simple_function_factory.h | 175 +++
be/src/vec/functions/time_of_function.cpp | 37 +
be/src/vec/functions/to_time_function.cpp | 54 +
be/src/vec/io/io_helper.h | 405 ++++++
.../io/reader_buffer.h} | 36 +-
be/src/vec/io/var_int.h | 176 +++
be/src/vec/olap/block_reader.cpp | 362 ++++++
be/src/vec/olap/block_reader.h | 120 ++
be/src/vec/olap/vcollect_iterator.cpp | 380 ++++++
be/src/vec/olap/vcollect_iterator.h | 193 +++
be/src/vec/olap/vgeneric_iterators.cpp | 443 +++++++
.../olap/vgeneric_iterators.h} | 8 +-
be/src/vec/runtime/vdata_stream_mgr.cpp | 194 +++
be/src/vec/runtime/vdata_stream_mgr.h | 91 ++
be/src/vec/runtime/vdata_stream_recvr.cpp | 364 ++++++
be/src/vec/runtime/vdata_stream_recvr.h | 188 +++
.../runtime/vdatetime_value.cpp} | 247 ++--
.../runtime/vdatetime_value.h} | 161 ++-
be/src/vec/runtime/vpartition_info.cpp | 56 +
be/src/vec/runtime/vpartition_info.h | 59 +
be/src/vec/runtime/vsorted_run_merger.cpp | 156 +++
be/src/vec/runtime/vsorted_run_merger.h | 91 ++
be/src/vec/sink/mysql_result_writer.cpp | 349 ++++++
be/src/{runtime => vec/sink}/mysql_result_writer.h | 48 +-
be/src/{runtime => vec/sink}/result_sink.cpp | 85 +-
be/src/vec/sink/result_sink.h | 79 ++
.../sink/result_writer.h} | 23 +-
be/src/vec/sink/vdata_stream_sender.cpp | 517 ++++++++
be/src/vec/sink/vdata_stream_sender.h | 265 ++++
be/src/vec/sink/vtabet_sink.cpp | 275 ++++
be/src/vec/sink/vtablet_sink.h | 65 +
be/src/vec/utils/util.hpp | 79 ++
be/test/exprs/string_functions_test.cpp | 43 +-
be/test/olap/column_reader_test.cpp | 108 ++
be/test/olap/generic_iterators_test.cpp | 6 +-
.../rowset/segment_v2/binary_dict_page_test.cpp | 2 +
.../segment_v2/column_reader_writer_test.cpp | 122 ++
be/test/tools/benchmark_tool.cpp | 2 +-
be/test/vec/aggregate_functions/CMakeLists.txt | 23 +
be/test/vec/aggregate_functions/agg_test.cpp | 61 +
be/test/vec/core/CMakeLists.txt | 23 +
be/test/vec/core/block_test.cpp | 309 +++++
be/test/vec/core/column_complex_test.cpp | 52 +
be/test/vec/exec/CMakeLists.txt | 21 +
be/test/vec/exec/vgeneric_iterators_test.cpp | 207 ++++
be/test/vec/exprs/CMakeLists.txt | 22 +
be/test/vec/exprs/vexpr_test.cpp | 357 ++++++
be/test/vec/function/CMakeLists.txt | 31 +
be/test/vec/function/function_abs_test.cpp | 92 ++
be/test/vec/function/function_arithmetic_test.cpp | 142 +++
be/test/vec/function/function_bitmap_test.cpp | 120 ++
be/test/vec/function/function_comparison_test.cpp | 150 +++
be/test/vec/function/function_hash_test.cpp | 103 ++
be/test/vec/function/function_ifnull_test.cpp | 85 ++
be/test/vec/function/function_like_test.cpp | 125 ++
be/test/vec/function/function_math_test.cpp | 425 +++++++
be/test/vec/function/function_nullif_test.cpp | 75 ++
be/test/vec/function/function_string_test.cpp | 690 +++++++++++
be/test/vec/function/function_test_util.h | 291 +++++
be/test/vec/function/function_time_test.cpp | 582 +++++++++
be/test/vec/runtime/CMakeLists.txt | 22 +
be/test/vec/runtime/vdata_stream_test.cpp | 178 +++
build.sh | 2 +-
.../org/apache/doris/analysis/AnalyticExpr.java | 6 +-
.../java/org/apache/doris/analysis/Analyzer.java | 25 +
.../org/apache/doris/analysis/ArithmeticExpr.java | 95 +-
.../org/apache/doris/analysis/BinaryPredicate.java | 2 +
.../org/apache/doris/analysis/InPredicate.java | 9 +-
.../org/apache/doris/analysis/LikePredicate.java | 5 +-
.../java/org/apache/doris/analysis/SelectStmt.java | 5 +
.../apache/doris/analysis/SetOperationStmt.java | 5 +-
.../java/org/apache/doris/analysis/TableRef.java | 6 +
.../apache/doris/catalog/AggregateFunction.java | 19 +-
.../java/org/apache/doris/catalog/Catalog.java | 17 +-
.../java/org/apache/doris/catalog/FunctionSet.java | 77 +-
.../main/java/org/apache/doris/catalog/Type.java | 25 +
.../doris/common/profile/PlanTreeBuilder.java | 1 +
.../doris/common/profile/ProfileTreeBuilder.java | 6 +
.../org/apache/doris/planner/AnalyticPlanner.java | 1 +
.../org/apache/doris/planner/ExchangeNode.java | 1 +
.../org/apache/doris/planner/HashJoinNode.java | 47 +-
.../org/apache/doris/planner/OlapScanNode.java | 1 -
.../java/org/apache/doris/planner/PlanNode.java | 9 +-
.../java/org/apache/doris/planner/Planner.java | 1 +
.../apache/doris/planner/SingleNodePlanner.java | 4 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 1 -
.../apache/doris/rewrite/FoldConstantsRule.java | 4 +-
gensrc/proto/palo_internal_service.proto | 46 +
gensrc/script/doris_builtins_functions.py | 255 ++--
gensrc/script/gen_build_version.sh | 2 +-
gensrc/script/gen_builtins_functions.py | 5 +
gensrc/thrift/DataSinks.thrift | 2 -
gensrc/thrift/PlanNodes.thrift | 5 +
471 files changed, 62946 insertions(+), 1396 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 24f28b1..1bd3e49 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -469,6 +469,7 @@ set(DORIS_LINK_LIBS
DorisGen
Webserver
Geo
+ Vec
Plugin
${WL_END_GROUP}
)
@@ -653,6 +654,7 @@ endif()
add_subdirectory(${SRC_DIR}/util)
add_subdirectory(${SRC_DIR}/plugin)
+add_subdirectory(${SRC_DIR}/vec)
# Utility CMake function to make specifying tests and benchmarks less verbose
FUNCTION(ADD_BE_TEST TEST_NAME)
@@ -705,6 +707,11 @@ if (${MAKE_TEST} STREQUAL "ON")
add_subdirectory(${TEST_DIR}/runtime)
add_subdirectory(${TEST_DIR}/udf)
add_subdirectory(${TEST_DIR}/util)
+ add_subdirectory(${TEST_DIR}/vec/core)
+ add_subdirectory(${TEST_DIR}/vec/exprs)
+ add_subdirectory(${TEST_DIR}/vec/function)
+ add_subdirectory(${TEST_DIR}/vec/runtime)
+ add_subdirectory(${TEST_DIR}/vec/aggregate_functions)
add_subdirectory(${TEST_DIR}/plugin)
add_subdirectory(${TEST_DIR}/plugin/example)
add_subdirectory(${TEST_DIR}/tools)
@@ -728,3 +735,8 @@ install(FILES
${BASE_DIR}/../conf/odbcinst.ini
DESTINATION ${OUTPUT_DIR}/conf)
+
+get_property(dirs DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY INCLUDE_DIRECTORIES)
+foreach(dir ${dirs})
+ message(STATUS "dir='${dir}'")
+ endforeach()
diff --git a/be/src/exec/blocking_join_node.cpp b/be/src/exec/blocking_join_node.cpp
index da97701..ba13786 100644
--- a/be/src/exec/blocking_join_node.cpp
+++ b/be/src/exec/blocking_join_node.cpp
@@ -30,7 +30,8 @@ namespace doris {
BlockingJoinNode::BlockingJoinNode(const std::string& node_name, const TJoinOp::type join_op,
ObjectPool* pool, const TPlanNode& tnode,
const DescriptorTbl& descs)
- : ExecNode(pool, tnode, descs), _node_name(node_name), _join_op(join_op) {}
+ : ExecNode(pool, tnode, descs), _node_name(node_name), _join_op(join_op),
+ _left_side_eos(false) {}
Status BlockingJoinNode::init(const TPlanNode& tnode, RuntimeState* state) {
return ExecNode::init(tnode, state);
diff --git a/be/src/exec/data_sink.cpp b/be/src/exec/data_sink.cpp
index aefa087..cf06f09 100644
--- a/be/src/exec/data_sink.cpp
+++ b/be/src/exec/data_sink.cpp
@@ -35,7 +35,10 @@
#include "runtime/result_file_sink.h"
#include "runtime/result_sink.h"
#include "runtime/runtime_state.h"
-#include "util/logging.h"
+
+#include "vec/sink/result_sink.h"
+#include "vec/sink/vdata_stream_sender.h"
+#include "vec/sink/vtablet_sink.h"
namespace doris {
@@ -57,6 +60,9 @@ Status DataSink::create_data_sink(ObjectPool* pool, const TDataSink& thrift_sink
: false;
// TODO: figure out good buffer size based on size of output row
if (is_vec) {
+ tmp_sink = new doris::vectorized::VDataStreamSender(
+ pool, params.sender_id, row_desc, thrift_sink.stream_sink, params.destinations,
+ 16 * 1024, send_query_statistics_with_every_batch);
} else {
tmp_sink = new DataStreamSender(pool, params.sender_id, row_desc,
thrift_sink.stream_sink, params.destinations, 16 * 1024,
@@ -73,6 +79,7 @@ Status DataSink::create_data_sink(ObjectPool* pool, const TDataSink& thrift_sink
// TODO: figure out good buffer size based on size of output row
if (is_vec) {
+ tmp_sink = new doris::vectorized::VResultSink(row_desc, output_exprs, thrift_sink.result_sink, 4096);
} else {
tmp_sink = new ResultSink(row_desc, output_exprs, thrift_sink.result_sink, 1024);
}
@@ -149,7 +156,11 @@ Status DataSink::create_data_sink(ObjectPool* pool, const TDataSink& thrift_sink
case TDataSinkType::OLAP_TABLE_SINK: {
Status status;
DCHECK(thrift_sink.__isset.olap_table_sink);
- sink->reset(new stream_load::OlapTableSink(pool, row_desc, output_exprs, &status));
+ if (is_vec) {
+ sink->reset(new stream_load::VOlapTableSink(pool, row_desc, output_exprs, &status));
+ } else {
+ sink->reset(new stream_load::OlapTableSink(pool, row_desc, output_exprs, &status));
+ }
RETURN_IF_ERROR(status);
break;
}
diff --git a/be/src/exec/data_sink.h b/be/src/exec/data_sink.h
index 30f5580..fcec10a 100644
--- a/be/src/exec/data_sink.h
+++ b/be/src/exec/data_sink.h
@@ -33,11 +33,13 @@ class ObjectPool;
class RowBatch;
class RuntimeProfile;
class RuntimeState;
-class TPlanExecRequest;
-class TPlanExecParams;
class TPlanFragmentExecParams;
class RowDescriptor;
+namespace vectorized {
+class Block;
+}
+
// Superclass of all data sinks.
class DataSink {
public:
@@ -56,8 +58,11 @@ public:
// Send a row batch into this sink.
// eos should be true when the last batch is passed to send()
virtual Status send(RuntimeState* state, RowBatch* batch) = 0;
- // virtual Status send(RuntimeState* state, RowBatch* batch, bool eos) = 0;
+ // Send a Block into this sink.
+ virtual Status send(RuntimeState* state, vectorized::Block* block) {
+ return Status::NotSupported("Not support send block");
+ };
// Releases all resources that were allocated in prepare()/send().
// Further send() calls are illegal after calling close().
// It must be okay to call this multiple times. Subsequent calls should
diff --git a/be/src/exec/es/es_scroll_parser.cpp b/be/src/exec/es/es_scroll_parser.cpp
index 396c086..86cd16a 100644
--- a/be/src/exec/es/es_scroll_parser.cpp
+++ b/be/src/exec/es/es_scroll_parser.cpp
@@ -31,6 +31,7 @@
#include "runtime/mem_pool.h"
#include "runtime/mem_tracker.h"
#include "util/string_parser.hpp"
+#include "vec/runtime/vdatetime_value.h"
namespace doris {
@@ -79,6 +80,8 @@ static const std::string ERROR_MEM_LIMIT_EXCEEDED =
static const std::string ERROR_COL_DATA_IS_ARRAY =
"Data source returned an array for the type $0"
"based on column metadata.";
+static const std::string INVALID_NULL_VALUE =
+ "Invalid null value occurs: Non-null column `$0` contains NULL";
#define RETURN_ERROR_IF_COL_IS_ARRAY(col, type) \
do { \
@@ -169,7 +172,7 @@ static Status get_int_value(const rapidjson::Value& col, PrimitiveType type, voi
template <typename T>
static Status get_float_value(const rapidjson::Value& col, PrimitiveType type, void* slot,
bool pure_doc_value) {
- DCHECK(sizeof(T) == 4 || sizeof(T) == 8);
+ static_assert(sizeof(T) == 4 || sizeof(T) == 8);
if (col.IsNumber()) {
*reinterpret_cast<T*>(slot) = (T)(sizeof(T) == 4 ? col.GetFloat() : col.GetDouble());
return Status::OK();
@@ -193,6 +196,68 @@ static Status get_float_value(const rapidjson::Value& col, PrimitiveType type, v
return Status::OK();
}
+template <typename T>
+static Status insert_float_value(const rapidjson::Value& col, PrimitiveType type,
+ vectorized::IColumn* col_ptr, bool pure_doc_value,
+ bool nullable) {
+ static_assert(sizeof(T) == 4 || sizeof(T) == 8);
+ if (col.IsNumber() && nullable) {
+ T value = (T)(sizeof(T) == 4 ? col.GetFloat() : col.GetDouble());
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&value)), 0);
+ return Status::OK();
+ }
+
+ if (pure_doc_value && col.IsArray() && nullable) {
+ T value = (T)(sizeof(T) == 4 ? col[0].GetFloat() : col[0].GetDouble());
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&value)), 0);
+ return Status::OK();
+ }
+
+ RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
+ RETURN_ERROR_IF_COL_IS_NOT_STRING(col, type);
+
+ StringParser::ParseResult result;
+ const std::string& val = col.GetString();
+ size_t len = col.GetStringLength();
+ T v = StringParser::string_to_float<T>(val.c_str(), len, &result);
+ RETURN_ERROR_IF_PARSING_FAILED(result, col, type);
+
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&v)), 0);
+
+ return Status::OK();
+}
+
+template <typename T>
+static Status insert_int_value(const rapidjson::Value& col, PrimitiveType type,
+ vectorized::IColumn* col_ptr, bool pure_doc_value,
+ bool nullable) {
+ if (col.IsNumber()) {
+ T value = (T)(sizeof(T) < 8 ? col.GetInt() : col.GetInt64());
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&value)), 0);
+ return Status::OK();
+ }
+
+ if (pure_doc_value && col.IsArray()) {
+ RETURN_ERROR_IF_COL_IS_NOT_NUMBER(col[0], type);
+ T value = (T)(sizeof(T) < 8 ? col[0].GetInt() : col[0].GetInt64());
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&value)), 0);
+ return Status::OK();
+ }
+
+ RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
+ RETURN_ERROR_IF_COL_IS_NOT_STRING(col, type);
+
+ StringParser::ParseResult result;
+ const std::string& val = col.GetString();
+ size_t len = col.GetStringLength();
+ T v = StringParser::string_to_int<T>(val.c_str(), len, &result);
+ RETURN_ERROR_IF_PARSING_FAILED(result, col, type);
+
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&v)), 0);
+
+ return Status::OK();
+}
+
ScrollParser::ScrollParser(bool doc_value_mode)
: _scroll_id(""), _size(0), _line_index(0), _doc_value_mode(doc_value_mode) {}
@@ -426,22 +491,53 @@ Status ScrollParser::fill_tuple(const TupleDescriptor* tuple_desc, Tuple* tuple,
*reinterpret_cast<int8_t*>(slot) = col.GetInt();
break;
}
- if (pure_doc_value && col.IsArray()) {
+
+ bool is_nested_str = false;
+ if (pure_doc_value && col.IsArray() && col[0].IsBool()) {
*reinterpret_cast<int8_t*>(slot) = col[0].GetBool();
break;
+ } else if (pure_doc_value && col.IsArray() && col[0].IsString()) {
+ is_nested_str = true;
+ } else if (pure_doc_value && col.IsArray()) {
+ return Status::InternalError(
+ strings::Substitute(ERROR_INVALID_COL_DATA, "BOOLEAN"));
}
- RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
- RETURN_ERROR_IF_COL_IS_NOT_STRING(col, type);
-
- const std::string& val = col.GetString();
- size_t val_size = col.GetStringLength();
+ const rapidjson::Value& str_col = is_nested_str? col[0]: col;
+ const std::string& val = str_col.GetString();
+ size_t val_size = str_col.GetStringLength();
StringParser::ParseResult result;
bool b = StringParser::string_to_bool(val.c_str(), val_size, &result);
- RETURN_ERROR_IF_PARSING_FAILED(result, col, type);
+ RETURN_ERROR_IF_PARSING_FAILED(result, str_col, type);
*reinterpret_cast<int8_t*>(slot) = b;
break;
}
+ case TYPE_DECIMALV2: {
+ DecimalV2Value data;
+
+ if (col.IsDouble()) {
+ data.assign_from_double(col.GetDouble());
+ } else {
+ std::string val;
+ if (pure_doc_value) {
+ if (!col[0].IsString()) {
+ val = json_value_to_string(col[0]);
+ } else {
+ val = col[0].GetString();
+ }
+ } else {
+ RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
+ if (!col.IsString()) {
+ val = json_value_to_string(col);
+ } else {
+ val = col.GetString();
+ }
+ }
+ data.parse_from_str(val.data(), val.length());
+ }
+ reinterpret_cast<DecimalV2Value*>(slot)->set_value(data.value());
+ break;
+ }
case TYPE_DATE:
case TYPE_DATETIME: {
@@ -482,6 +578,235 @@ Status ScrollParser::fill_tuple(const TupleDescriptor* tuple_desc, Tuple* tuple,
return Status::OK();
}
+Status ScrollParser::fill_columns(const TupleDescriptor* tuple_desc,
+ std::vector<vectorized::MutableColumnPtr>& columns,
+ MemPool* tuple_pool, bool* line_eof,
+ const std::map<std::string, std::string>& docvalue_context) {
+ *line_eof = true;
+
+ if (_size <= 0 || _line_index >= _size) {
+ return Status::OK();
+ }
+
+ const rapidjson::Value& obj = _inner_hits_node[_line_index++];
+ bool pure_doc_value = false;
+ if (obj.HasMember("fields")) {
+ pure_doc_value = true;
+ }
+ const rapidjson::Value& line = obj.HasMember(FIELD_SOURCE) ? obj[FIELD_SOURCE] : obj["fields"];
+
+ for (int i = 0; i < tuple_desc->slots().size(); ++i) {
+ const SlotDescriptor* slot_desc = tuple_desc->slots()[i];
+ auto col_ptr = columns[i].get();
+
+ if (!slot_desc->is_materialized()) {
+ continue;
+ }
+ if (slot_desc->col_name() == FIELD_ID) {
+ // actually this branch will not be reached, this is guaranteed by Doris FE.
+ if (pure_doc_value) {
+ std::stringstream ss;
+ ss << "obtain `_id` is not supported in doc_values mode";
+ return Status::RuntimeError(ss.str());
+ }
+ // obj[FIELD_ID] must not be NULL
+ std::string _id = obj[FIELD_ID].GetString();
+ size_t len = _id.length();
+
+ col_ptr->insert_data(const_cast<const char*>(_id.data()), len);
+ continue;
+ }
+
+ const char* col_name = pure_doc_value ? docvalue_context.at(slot_desc->col_name()).c_str()
+ : slot_desc->col_name().c_str();
+
+ rapidjson::Value::ConstMemberIterator itr = line.FindMember(col_name);
+ if (itr == line.MemberEnd() && slot_desc->is_nullable()) {
+ auto nullable_column = reinterpret_cast<vectorized::ColumnNullable*>(col_ptr);
+ nullable_column->insert_data(nullptr, 0);
+ continue;
+ } else if (itr == line.MemberEnd() && !slot_desc->is_nullable()) {
+ std::string details = strings::Substitute(INVALID_NULL_VALUE, col_name);
+ return Status::RuntimeError(details);
+ }
+
+ const rapidjson::Value& col = line[col_name];
+
+ PrimitiveType type = slot_desc->type().type;
+
+ // when the column value is null, the subsequent type casting will report an error
+ if (col.IsNull() && slot_desc->is_nullable()) {
+ col_ptr->insert_data(nullptr, 0);
+ continue;
+ } else if (col.IsNull() && !slot_desc->is_nullable()) {
+ std::string details = strings::Substitute(INVALID_NULL_VALUE, col_name);
+ return Status::RuntimeError(details);
+ }
+ switch (type) {
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING: {
+ // sometimes elasticsearch user post some not-string value to Elasticsearch Index.
+ // because of reading value from _source, we can not process all json type and then just transfer the value to original string representation
+ // this may be a tricky, but we can workaround this issue
+ std::string val;
+ if (pure_doc_value) {
+ if (!col[0].IsString()) {
+ val = json_value_to_string(col[0]);
+ } else {
+ val = col[0].GetString();
+ }
+ } else {
+ RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
+ if (!col.IsString()) {
+ val = json_value_to_string(col);
+ } else {
+ val = col.GetString();
+ }
+ }
+ size_t val_size = val.length();
+ col_ptr->insert_data(
+ const_cast<const char*>(val.data()), val_size);
+ break;
+ }
+
+ case TYPE_TINYINT: {
+ insert_int_value<int8_t>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_SMALLINT: {
+ insert_int_value<int16_t>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_INT: {
+ insert_int_value<int32>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_BIGINT: {
+ insert_int_value<int64_t>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_LARGEINT: {
+ insert_int_value<__int128>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_DOUBLE: {
+ insert_float_value<double>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_FLOAT: {
+ insert_float_value<float>(col, type, col_ptr, pure_doc_value, slot_desc->is_nullable());
+ break;
+ }
+
+ case TYPE_BOOLEAN: {
+ if (col.IsBool()) {
+ int8_t val = col.GetBool();
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&val)), 0);
+ break;
+ }
+
+ if (col.IsNumber()) {
+ int8_t val = col.GetInt();
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&val)), 0);
+ break;
+ }
+
+ bool is_nested_str = false;
+ if (pure_doc_value && col.IsArray() && col[0].IsBool()) {
+ int8_t val = col[0].GetBool();
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&val)), 0);
+ break;
+ } else if (pure_doc_value && col.IsArray() && col[0].IsString()) {
+ is_nested_str = true;
+ } else if (pure_doc_value && col.IsArray()) {
+ return Status::InternalError(
+ strings::Substitute(ERROR_INVALID_COL_DATA, "BOOLEAN"));
+ }
+
+ const rapidjson::Value& str_col = is_nested_str? col[0]: col;
+
+ const std::string& val = str_col.GetString();
+ size_t val_size = str_col.GetStringLength();
+ StringParser::ParseResult result;
+ bool b = StringParser::string_to_bool(val.c_str(), val_size, &result);
+ RETURN_ERROR_IF_PARSING_FAILED(result, str_col, type);
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&b)), 0);
+ break;
+ }
+ case TYPE_DECIMALV2: {
+ DecimalV2Value data;
+
+ if (col.IsDouble()) {
+ data.assign_from_double(col.GetDouble());
+ } else {
+ std::string val;
+ if (pure_doc_value) {
+ if (!col[0].IsString()) {
+ val = json_value_to_string(col[0]);
+ } else {
+ val = col[0].GetString();
+ }
+ } else {
+ RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
+ if (!col.IsString()) {
+ val = json_value_to_string(col);
+ } else {
+ val = col.GetString();
+ }
+ }
+ data.parse_from_str(val.data(), val.length());
+ }
+ col_ptr->insert_data(
+ const_cast<const char*>(reinterpret_cast<char*>(&data)), 0);
+ break;
+ }
+
+ case TYPE_DATE:
+ case TYPE_DATETIME: {
+ // this would happend just only when `enable_docvalue_scan = false`, and field has timestamp format date from _source
+ if (col.IsNumber()) {
+ // ES process date/datetime field would use millisecond timestamp for index or docvalue
+ // processing date type field, if a number is encountered, Doris On ES will force it to be processed according to ms
+ // Doris On ES needs to be consistent with ES, so just divided by 1000 because the unit for from_unixtime is seconds
+ RETURN_IF_ERROR(fill_date_col_with_timestamp(col_ptr, col, type));
+ } else if (col.IsArray() && pure_doc_value) {
+ // this would happened just only when `enable_docvalue_scan = true`
+ // ES add default format for all field after ES 6.4, if we not provided format for `date` field ES would impose
+ // a standard date-format for date field as `2020-06-16T00:00:00.000Z`
+ // At present, we just process this string format date. After some PR were merged into Doris, we would impose `epoch_mills` for
+ // date field's docvalue
+ if (col[0].IsString()) {
+ RETURN_IF_ERROR(fill_date_col_with_strval(col_ptr, col[0], type));
+ break;
+ }
+ // ES would return millisecond timestamp for date field, divided by 1000 because the unit for from_unixtime is seconds
+ RETURN_IF_ERROR(fill_date_col_with_timestamp(col_ptr, col, type));
+ } else {
+ // this would happened just only when `enable_docvalue_scan = false`, and field has string format date from _source
+ RETURN_ERROR_IF_COL_IS_ARRAY(col, type);
+ RETURN_ERROR_IF_COL_IS_NOT_STRING(col, type);
+ RETURN_IF_ERROR(fill_date_col_with_strval(col_ptr, col, type));
+ }
+ break;
+ }
+ default: {
+ DCHECK(false);
+ break;
+ }
+ }
+ }
+
+ *line_eof = false;
+ return Status::OK();
+}
+
Status ScrollParser::fill_date_slot_with_strval(void* slot, const rapidjson::Value& col,
PrimitiveType type) {
DateTimeValue* ts_slot = reinterpret_cast<DateTimeValue*>(slot);
@@ -511,4 +836,44 @@ Status ScrollParser::fill_date_slot_with_timestamp(void* slot, const rapidjson::
return Status::OK();
}
+Status ScrollParser::fill_date_col_with_strval(vectorized::IColumn* col_ptr,
+ const rapidjson::Value& col, PrimitiveType type) {
+ vectorized::VecDateTimeValue dt_val;
+ const std::string& val = col.GetString();
+ size_t val_size = col.GetStringLength();
+ if (!dt_val.from_date_str(val.c_str(), val_size)) {
+ RETURN_ERROR_IF_CAST_FORMAT_ERROR(col, type);
+ }
+ if (type == TYPE_DATE) {
+ dt_val.cast_to_date();
+ } else {
+ dt_val.to_datetime();
+ }
+
+ auto date_packed_int = binary_cast<doris::vectorized::VecDateTimeValue, int64_t>(
+ *reinterpret_cast<vectorized::VecDateTimeValue*>(&dt_val));
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&date_packed_int)), 0);
+ return Status::OK();
+}
+
+Status ScrollParser::fill_date_col_with_timestamp(vectorized::IColumn* col_ptr,
+ const rapidjson::Value& col, PrimitiveType type) {
+ vectorized::VecDateTimeValue dt_val;
+ if (!dt_val.from_unixtime(col.GetInt64() / 1000, "+08:00")) {
+ RETURN_ERROR_IF_CAST_FORMAT_ERROR(col, type);
+ }
+ if (type == TYPE_DATE) {
+ reinterpret_cast<vectorized::VecDateTimeValue*>(&dt_val)->cast_to_date();
+ } else {
+ reinterpret_cast<vectorized::VecDateTimeValue*>(&dt_val)->set_type(TIME_DATETIME);
+ }
+
+ auto date_packed_int = binary_cast<doris::vectorized::VecDateTimeValue, int64_t>(
+ *reinterpret_cast<vectorized::VecDateTimeValue*>(&dt_val));
+ col_ptr->insert_data(const_cast<const char*>(reinterpret_cast<char*>(&date_packed_int)), 0);
+
+ return Status::OK();
+}
+
+
} // namespace doris
diff --git a/be/src/exec/es/es_scroll_parser.h b/be/src/exec/es/es_scroll_parser.h
index f2ff855..30eabaf 100644
--- a/be/src/exec/es/es_scroll_parser.h
+++ b/be/src/exec/es/es_scroll_parser.h
@@ -22,6 +22,7 @@
#include "rapidjson/document.h"
#include "runtime/descriptors.h"
#include "runtime/tuple.h"
+#include "vec/core/block.h"
namespace doris {
@@ -35,6 +36,9 @@ public:
Status parse(const std::string& scroll_result, bool exactly_once = false);
Status fill_tuple(const TupleDescriptor* _tuple_desc, Tuple* tuple, MemPool* mem_pool,
bool* line_eof, const std::map<std::string, std::string>& docvalue_context);
+ Status fill_columns(const TupleDescriptor* _tuple_desc,
+ std::vector<vectorized::MutableColumnPtr>& columns, MemPool* mem_pool,
+ bool* line_eof, const std::map<std::string, std::string>& docvalue_context);
const std::string& get_scroll_id();
int get_size();
@@ -44,9 +48,13 @@ private:
// type is used for distinguish date and datetime
// fill date slot with string format date
Status fill_date_slot_with_strval(void* slot, const rapidjson::Value& col, PrimitiveType type);
+ Status fill_date_col_with_strval(vectorized::IColumn* col_ptr, const rapidjson::Value& col,
+ PrimitiveType type);
// fill date slot with timestamp
Status fill_date_slot_with_timestamp(void* slot, const rapidjson::Value& col,
PrimitiveType type);
+ Status fill_date_col_with_timestamp(vectorized::IColumn* col_ptr, const rapidjson::Value& col,
+ PrimitiveType type);
private:
std::string _scroll_id;
diff --git a/be/src/exec/es_http_scan_node.cpp b/be/src/exec/es_http_scan_node.cpp
index 012110d..7b67486 100644
--- a/be/src/exec/es_http_scan_node.cpp
+++ b/be/src/exec/es_http_scan_node.cpp
@@ -437,10 +437,17 @@ void EsHttpScanNode::scanner_worker(int start_idx, int length, std::promise<Stat
properties, _column_names, _predicates, _docvalue_context, &doc_value_mode);
// start scanner to scan
- std::unique_ptr<EsHttpScanner> scanner(
- new EsHttpScanner(_runtime_state, runtime_profile(), _tuple_id, properties,
- scanner_expr_ctxs, &counter, doc_value_mode));
- status = scanner_scan(std::move(scanner), scanner_expr_ctxs, &counter);
+ if (!_vectorized) {
+ std::unique_ptr<EsHttpScanner> scanner(
+ new EsHttpScanner(_runtime_state, runtime_profile(), _tuple_id, properties,
+ scanner_expr_ctxs, &counter, doc_value_mode));
+ status = scanner_scan(std::move(scanner), scanner_expr_ctxs, &counter);
+ } else {
+ std::unique_ptr<VEsHttpScanner> scanner(
+ new VEsHttpScanner(_runtime_state, runtime_profile(), _tuple_id, properties,
+ scanner_expr_ctxs, &counter, doc_value_mode));
+ status = scanner_scan(std::move(scanner));
+ }
if (!status.ok()) {
LOG(WARNING) << "Scanner[" << start_idx
<< "] process failed. status=" << status.get_error_msg();
diff --git a/be/src/exec/es_http_scan_node.h b/be/src/exec/es_http_scan_node.h
index 640ea0c..b90bf0e 100644
--- a/be/src/exec/es_http_scan_node.h
+++ b/be/src/exec/es_http_scan_node.h
@@ -29,6 +29,7 @@
#include "common/status.h"
#include "exec/es_http_scanner.h"
+#include "vec/exec/ves_http_scanner.h"
#include "exec/scan_node.h"
#include "gen_cpp/PaloInternalService_types.h"
@@ -56,7 +57,6 @@ protected:
// Write debug string of this into out.
virtual void debug_string(int indentation_level, std::stringstream* out) const override;
-private:
// Update process status to one failed status,
// NOTE: Must hold the mutex of this scan node
bool update_status(const Status& new_status) {
@@ -67,20 +67,8 @@ private:
return false;
}
- // Create scanners to do scan job
- Status start_scanners();
-
- // Collect all scanners 's status
- Status collect_scanners_status();
-
// One scanner worker, This scanner will handle 'length' ranges start from start_idx
- void scanner_worker(int start_idx, int length, std::promise<Status>& p_status);
-
- // Scan one range
- Status scanner_scan(std::unique_ptr<EsHttpScanner> scanner,
- const std::vector<ExprContext*>& conjunct_ctxs, EsScanCounter* counter);
-
- Status build_conjuncts_list();
+ virtual void scanner_worker(int start_idx, int length, std::promise<Status>& p_status);
TupleId _tuple_id;
RuntimeState* _runtime_state;
@@ -92,20 +80,41 @@ private:
int _max_buffered_batches;
RuntimeProfile::Counter* _wait_scanner_timer;
- bool _all_scanners_finished;
Status _process_status;
+ std::map<std::string, std::string> _docvalue_context;
+
+ std::condition_variable _queue_reader_cond;
+ std::condition_variable _queue_writer_cond;
+ bool _vectorized = false;
+
+private:
+ // Create scanners to do scan job
+ Status start_scanners();
+
+ // Collect all scanners 's status
+ Status collect_scanners_status();
+
+ // Scan one range
+ Status scanner_scan(std::unique_ptr<EsHttpScanner> scanner,
+ const std::vector<ExprContext*>& conjunct_ctxs, EsScanCounter* counter);
+
+ virtual Status scanner_scan(std::unique_ptr<VEsHttpScanner> scanner) {
+ return Status::NotSupported("vectorized scan in EsHttpScanNode is not supported!");
+ };
+
+ Status build_conjuncts_list();
+
+ bool _all_scanners_finished;
+
std::vector<std::thread> _scanner_threads;
std::vector<std::promise<Status>> _scanners_status;
std::map<std::string, std::string> _properties;
- std::map<std::string, std::string> _docvalue_context;
std::map<std::string, std::string> _fields_context;
std::vector<TScanRangeParams> _scan_ranges;
std::vector<std::string> _column_names;
std::mutex _batch_queue_lock;
- std::condition_variable _queue_reader_cond;
- std::condition_variable _queue_writer_cond;
std::deque<std::shared_ptr<RowBatch>> _batch_queue;
std::vector<EsPredicate*> _predicates;
diff --git a/be/src/exec/es_http_scanner.h b/be/src/exec/es_http_scanner.h
index 2bade5a..dcebfe1 100644
--- a/be/src/exec/es_http_scanner.h
+++ b/be/src/exec/es_http_scanner.h
@@ -68,7 +68,7 @@ public:
void close();
-private:
+protected:
RuntimeState* _state;
RuntimeProfile* _profile;
TupleId _tuple_id;
diff --git a/be/src/exec/exec_node.cpp b/be/src/exec/exec_node.cpp
index d692a79..b18160e 100644
--- a/be/src/exec/exec_node.cpp
+++ b/be/src/exec/exec_node.cpp
@@ -63,6 +63,25 @@
#include "util/debug_util.h"
#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+#include "vec/exec/join/vhash_join_node.h"
+#include "vec/exec/vaggregation_node.h"
+#include "vec/exec/ves_http_scan_node.h"
+#include "vec/exec/vcross_join_node.h"
+#include "vec/exec/vexchange_node.h"
+#include "vec/exec/vmysql_scan_node.h"
+#include "vec/exec/vodbc_scan_node.h"
+#include "vec/exec/volap_scan_node.h"
+#include "vec/exec/vsort_node.h"
+#include "vec/exec/vunion_node.h"
+#include "vec/exec/vintersect_node.h"
+#include "vec/exec/vexcept_node.h"
+#include "vec/exec/vanalytic_eval_node.h"
+#include "vec/exec/vassert_num_rows_node.h"
+#include "vec/exec/vselect_node.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exec/vempty_set_node.h"
+#include "vec/exec/vschema_scan_node.h"
namespace doris {
const std::string ExecNode::ROW_THROUGHPUT_COUNTER = "RowsReturnedRate";
@@ -167,6 +186,9 @@ Status ExecNode::init(const TPlanNode& tnode, RuntimeState* state) {
init_runtime_profile(profile);
if (tnode.__isset.vconjunct) {
+ _vconjunct_ctx_ptr.reset(new doris::vectorized::VExprContext*);
+ RETURN_IF_ERROR(doris::vectorized::VExpr::create_expr_tree(_pool, tnode.vconjunct,
+ _vconjunct_ctx_ptr.get()));
}
RETURN_IF_ERROR(Expr::create_expr_trees(_pool, tnode.conjuncts, &_conjunct_ctxs));
@@ -189,6 +211,9 @@ Status ExecNode::prepare(RuntimeState* state) {
_mem_tracker);
_expr_mem_pool.reset(new MemPool(_expr_mem_tracker.get()));
+ if (_vconjunct_ctx_ptr) {
+ RETURN_IF_ERROR((*_vconjunct_ctx_ptr)->prepare(state, row_desc(), expr_mem_tracker()));
+ }
RETURN_IF_ERROR(Expr::prepare(_conjunct_ctxs, state, row_desc(), expr_mem_tracker()));
// TODO(zc):
@@ -202,6 +227,9 @@ Status ExecNode::prepare(RuntimeState* state) {
Status ExecNode::open(RuntimeState* state) {
RETURN_IF_ERROR(exec_debug_action(TExecNodePhase::OPEN));
+ if (_vconjunct_ctx_ptr) {
+ RETURN_IF_ERROR((*_vconjunct_ctx_ptr)->open(state));
+ }
return Expr::open(_conjunct_ctxs, state);
}
@@ -240,6 +268,7 @@ Status ExecNode::close(RuntimeState* state) {
}
}
+ if (_vconjunct_ctx_ptr) (*_vconjunct_ctx_ptr)->close(state);
Expr::close(_conjunct_ctxs, state);
if (expr_mem_pool() != nullptr) {
@@ -339,12 +368,41 @@ Status ExecNode::create_tree_helper(RuntimeState* state, ObjectPool* pool,
Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanNode& tnode,
const DescriptorTbl& descs, ExecNode** node) {
+ std::stringstream error_msg;
+
if (state->enable_vectorized_exec()) {
- return Status::InternalError("unsupport enable_vectorized_engine");
+ switch (tnode.node_type) {
+ case TPlanNodeType::OLAP_SCAN_NODE:
+ case TPlanNodeType::ASSERT_NUM_ROWS_NODE:
+ case TPlanNodeType::HASH_JOIN_NODE:
+ case TPlanNodeType::AGGREGATION_NODE:
+ case TPlanNodeType::UNION_NODE:
+ case TPlanNodeType::CROSS_JOIN_NODE:
+ case TPlanNodeType::SORT_NODE:
+ case TPlanNodeType::EXCHANGE_NODE:
+ case TPlanNodeType::ODBC_SCAN_NODE:
+ case TPlanNodeType::MYSQL_SCAN_NODE:
+ case TPlanNodeType::INTERSECT_NODE:
+ case TPlanNodeType::EXCEPT_NODE:
+ case TPlanNodeType::ES_HTTP_SCAN_NODE:
+ case TPlanNodeType::EMPTY_SET_NODE:
+ case TPlanNodeType::SCHEMA_SCAN_NODE:
+ case TPlanNodeType::ANALYTIC_EVAL_NODE:
+ case TPlanNodeType::SELECT_NODE:
+ break;
+ default: {
+ const auto& i = _TPlanNodeType_VALUES_TO_NAMES.find(tnode.node_type);
+ const char* str = "unknown node type";
+
+ if (i != _TPlanNodeType_VALUES_TO_NAMES.end()) {
+ str = i->second;
+ }
+ error_msg << "V" << str << " not implemented";
+ return Status::InternalError(error_msg.str());
+ }
+ }
}
- std::stringstream error_msg;
-
VLOG_CRITICAL << "tnode:\n" << apache::thrift::ThriftDebugString(tnode);
switch (tnode.node_type) {
case TPlanNodeType::CSV_SCAN_NODE:
@@ -353,14 +411,20 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::MYSQL_SCAN_NODE:
#ifdef DORIS_WITH_MYSQL
- *node = pool->add(new MysqlScanNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VMysqlScanNode(pool, tnode, descs));
+ } else
+ *node = pool->add(new MysqlScanNode(pool, tnode, descs));
return Status::OK();
#else
return Status::InternalError(
"Don't support MySQL table, you should rebuild Doris with WITH_MYSQL option ON");
#endif
case TPlanNodeType::ODBC_SCAN_NODE:
- *node = pool->add(new OdbcScanNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VOdbcScanNode(pool, tnode, descs));
+ } else
+ *node = pool->add(new OdbcScanNode(pool, tnode, descs));
return Status::OK();
case TPlanNodeType::ES_SCAN_NODE:
@@ -368,15 +432,24 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
return Status::OK();
case TPlanNodeType::ES_HTTP_SCAN_NODE:
- *node = pool->add(new EsHttpScanNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VEsHttpScanNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new EsHttpScanNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::SCHEMA_SCAN_NODE:
- *node = pool->add(new SchemaScanNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VSchemaScanNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new SchemaScanNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::OLAP_SCAN_NODE:
if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VOlapScanNode(pool, tnode, descs));
} else {
*node = pool->add(new OlapScanNode(pool, tnode, descs));
}
@@ -384,6 +457,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::AGGREGATION_NODE:
if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::AggregationNode(pool, tnode, descs));
} else {
if (config::enable_partitioned_aggregation) {
*node = pool->add(new PartitionedAggregationNode(pool, tnode, descs));
@@ -394,11 +468,16 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
return Status::OK();
case TPlanNodeType::HASH_JOIN_NODE:
- *node = pool->add(new HashJoinNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::HashJoinNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new HashJoinNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::CROSS_JOIN_NODE:
if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VCrossJoinNode(pool, tnode, descs));
} else {
*node = pool->add(new CrossJoinNode(pool, tnode, descs));
}
@@ -409,18 +488,27 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
return Status::OK();
case TPlanNodeType::EMPTY_SET_NODE:
- *node = pool->add(new EmptySetNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VEmptySetNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new EmptySetNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::EXCHANGE_NODE:
if (state->enable_vectorized_exec()) {
+ *node = pool->add(new doris::vectorized::VExchangeNode(pool, tnode, descs));
} else {
*node = pool->add(new ExchangeNode(pool, tnode, descs));
}
return Status::OK();
case TPlanNodeType::SELECT_NODE:
- *node = pool->add(new SelectNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new doris::vectorized::VSelectNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new SelectNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::OLAP_REWRITE_NODE:
@@ -429,6 +517,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::SORT_NODE:
if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VSortNode(pool, tnode, descs));
} else {
if (tnode.sort_node.use_top_n) {
*node = pool->add(new TopNNode(pool, tnode, descs));
@@ -439,8 +528,12 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
return Status::OK();
case TPlanNodeType::ANALYTIC_EVAL_NODE:
- *node = pool->add(new AnalyticEvalNode(pool, tnode, descs));
- break;
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VAnalyticEvalNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new AnalyticEvalNode(pool, tnode, descs));
+ }
+ return Status::OK();
case TPlanNodeType::MERGE_NODE:
*node = pool->add(new MergeNode(pool, tnode, descs));
@@ -448,17 +541,26 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::UNION_NODE:
if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VUnionNode(pool, tnode, descs));
} else {
*node = pool->add(new UnionNode(pool, tnode, descs));
}
return Status::OK();
case TPlanNodeType::INTERSECT_NODE:
- *node = pool->add(new IntersectNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VIntersectNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new IntersectNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::EXCEPT_NODE:
- *node = pool->add(new ExceptNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VExceptNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new ExceptNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::BROKER_SCAN_NODE:
@@ -470,7 +572,11 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
return Status::OK();
case TPlanNodeType::ASSERT_NUM_ROWS_NODE:
- *node = pool->add(new AssertNumRowsNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VAssertNumRowsNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new AssertNumRowsNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::TABLE_FUNCTION_NODE:
@@ -637,6 +743,22 @@ Status ExecNode::claim_buffer_reservation(RuntimeState* state) {
Status ExecNode::release_unused_reservation() {
return _buffer_pool_client.DecreaseReservationTo(_resource_profile.min_reservation);
}
+
+void ExecNode::release_block_memory(vectorized::Block& block, uint16_t child_idx) {
+ DCHECK(child_idx < _children.size());
+ block.clear_column_data(child(child_idx)->row_desc().num_materialized_slots());
+}
+
+void ExecNode::reached_limit(vectorized::Block* block, bool* eos) {
+ if (_limit != -1 and _num_rows_returned + block->rows() >= _limit) {
+ block->set_num_rows(_limit - _num_rows_returned);
+ *eos = true;
+ }
+
+ _num_rows_returned += block->rows();
+ if (*eos) COUNTER_SET(_rows_returned_counter, _num_rows_returned);
+}
+
/*
Status ExecNode::enable_deny_reservation_debug_action() {
DCHECK_EQ(debug_action_, TDebugAction::SET_DENY_RESERVATION_PROBABILITY);
diff --git a/be/src/exec/exec_node.h b/be/src/exec/exec_node.h
index c58f862..7cad500 100644
--- a/be/src/exec/exec_node.h
+++ b/be/src/exec/exec_node.h
@@ -33,6 +33,8 @@
#include "util/runtime_profile.h"
#include "util/uid_util.h" // for print_id
+#include "vec/exprs/vexpr_context.h"
+
namespace doris {
class Expr;
class ExprContext;
@@ -219,6 +221,15 @@ protected:
/// fails.
Status release_unused_reservation();
+ /// Release all memory of block which got from child. The block
+ // 1. clear mem of valid column get from child, make sure child can reuse the mem
+ // 2. delete and release the column which create by function all and other reason
+ void release_block_memory(vectorized::Block& block, uint16_t child_idx = 0);
+
+ /// Only use in vectorized exec engine to check whether reach limit and cut num row for block
+ // and add block rows for profile
+ void reached_limit(vectorized::Block* block, bool* eos);
+
/// Enable the increase reservation denial probability on 'buffer_pool_client_' based on
/// the 'debug_action_' set on this node. Returns an error if 'debug_action_param_' is
/// invalid.
@@ -275,6 +286,8 @@ protected:
std::vector<ExprContext*> _conjunct_ctxs;
std::vector<TupleId> _tuple_ids;
+ std::unique_ptr<doris::vectorized::VExprContext*> _vconjunct_ctx_ptr;
+
std::vector<ExecNode*> _children;
RowDescriptor _row_descriptor;
diff --git a/be/src/exec/mysql_scan_node.h b/be/src/exec/mysql_scan_node.h
index 6d9d602..ff5fb54 100644
--- a/be/src/exec/mysql_scan_node.h
+++ b/be/src/exec/mysql_scan_node.h
@@ -63,7 +63,7 @@ private:
// The Mysql value is converted into the appropriate target type.
Status write_text_slot(char* value, int value_length, SlotDescriptor* slot,
RuntimeState* state);
-
+protected:
bool _is_init;
MysqlScannerParam _my_param;
// Name of Mysql table
diff --git a/be/src/exec/odbc_scan_node.cpp b/be/src/exec/odbc_scan_node.cpp
index 6bdc620..ea32cd8 100644
--- a/be/src/exec/odbc_scan_node.cpp
+++ b/be/src/exec/odbc_scan_node.cpp
@@ -29,9 +29,11 @@
namespace doris {
-OdbcScanNode::OdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs)
+OdbcScanNode::OdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs,
+ std::string scan_node_type)
: ScanNode(pool, tnode, descs),
_is_init(false),
+ _scan_node_type(scan_node_type),
_table_name(tnode.odbc_scan_node.table_name),
_connect_string(std::move(tnode.odbc_scan_node.connect_string)),
_query_string(std::move(tnode.odbc_scan_node.query_string)),
@@ -42,7 +44,7 @@ OdbcScanNode::OdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const Descr
OdbcScanNode::~OdbcScanNode() {}
Status OdbcScanNode::prepare(RuntimeState* state) {
- VLOG_CRITICAL << "OdbcScanNode::Prepare";
+ VLOG_CRITICAL << _scan_node_type << "::Prepare";
if (_is_init) {
return Status::OK();
@@ -91,7 +93,7 @@ Status OdbcScanNode::prepare(RuntimeState* state) {
Status OdbcScanNode::open(RuntimeState* state) {
RETURN_IF_ERROR(ExecNode::open(state));
- VLOG_CRITICAL << "OdbcScanNode::Open";
+ VLOG_CRITICAL << _scan_node_type << "::Open";
if (nullptr == state) {
return Status::InternalError("input pointer is null.");
@@ -125,7 +127,7 @@ Status OdbcScanNode::write_text_slot(char* value, int value_length, SlotDescript
}
Status OdbcScanNode::get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) {
- VLOG_CRITICAL << "OdbcScanNode::GetNext";
+ VLOG_CRITICAL << _scan_node_type << "::GetNext";
if (nullptr == state || nullptr == row_batch || nullptr == eos) {
return Status::InternalError("input is nullptr pointer");
@@ -240,7 +242,7 @@ Status OdbcScanNode::close(RuntimeState* state) {
void OdbcScanNode::debug_string(int indentation_level, std::stringstream* out) const {
*out << string(indentation_level * 2, ' ');
- *out << "OdbcScanNode(tupleid=" << _tuple_id << " table=" << _table_name;
+ *out << _scan_node_type << "(tupleid=" << _tuple_id << " table=" << _table_name;
*out << ")" << std::endl;
for (int i = 0; i < _children.size(); ++i) {
diff --git a/be/src/exec/odbc_scan_node.h b/be/src/exec/odbc_scan_node.h
index 393d9ac..1754993 100644
--- a/be/src/exec/odbc_scan_node.h
+++ b/be/src/exec/odbc_scan_node.h
@@ -35,7 +35,8 @@ class Status;
class OdbcScanNode : public ScanNode {
public:
- OdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs);
+ OdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs,
+ std::string scan_node_type = "OdbcScanNode");
~OdbcScanNode();
// initialize _odbc_scanner, and create _text_converter.
@@ -53,6 +54,12 @@ public:
// No use
virtual Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges);
+ const TupleDescriptor* get_tuple_desc() { return _tuple_desc; }
+ TextConverter* get_text_converter() { return _text_converter.get(); }
+ ODBCConnector* get_odbc_scanner() { return _odbc_scanner.get(); }
+ const std::string& get_scan_node_type() { return _scan_node_type; }
+
+ bool is_init() { return _is_init; }
protected:
// Write debug string of this into out.
@@ -65,6 +72,9 @@ private:
RuntimeState* state);
bool _is_init;
+
+ std::string _scan_node_type;
+
// Name of Odbc table
std::string _table_name;
diff --git a/be/src/exec/olap_scan_node.cpp b/be/src/exec/olap_scan_node.cpp
index ebf8720..dfc9a29 100644
--- a/be/src/exec/olap_scan_node.cpp
+++ b/be/src/exec/olap_scan_node.cpp
@@ -61,7 +61,7 @@ OlapScanNode::OlapScanNode(ObjectPool* pool, const TPlanNode& tnode, const Descr
Status OlapScanNode::init(const TPlanNode& tnode, RuntimeState* state) {
RETURN_IF_ERROR(ExecNode::init(tnode, state));
- _direct_conjunct_size = state->enable_vectorized_exec() ? 1 : _conjunct_ctxs.size();
+ _direct_conjunct_size = _conjunct_ctxs.size();
const TQueryOptions& query_options = state->query_options();
if (query_options.__isset.max_scan_key_num) {
@@ -464,7 +464,6 @@ Status OlapScanNode::start_scan(RuntimeState* state) {
VLOG_CRITICAL << "Filter idle conjuncts";
// 5. Filter idle conjunct which already trans to olap filters
// this must be after build_scan_key, it will free the StringValue memory
- // TODO: filter idle conjunct in vexpr_contexts
remove_pushed_conjuncts(state);
VLOG_CRITICAL << "StartScanThread";
@@ -516,6 +515,7 @@ void OlapScanNode::remove_pushed_conjuncts(RuntimeState* state) {
_conjunct_ctxs = std::move(new_conjunct_ctxs);
_direct_conjunct_size = new_direct_conjunct_size;
+ // TODO: support vbloom_filter_predicate/vbinary_predicate and merge unpushed predicate to _vconjunct_ctx
for (auto push_down_ctx : _pushed_conjuncts_index) {
auto iter = _conjunctid_to_runtime_filter_ctxs.find(push_down_ctx);
if (iter != _conjunctid_to_runtime_filter_ctxs.end()) {
@@ -524,7 +524,13 @@ void OlapScanNode::remove_pushed_conjuncts(RuntimeState* state) {
}
// set vconjunct_ctx is empty, if all conjunct
if (_direct_conjunct_size == 0) {
+ if (_vconjunct_ctx_ptr.get() != nullptr) {
+ (*_vconjunct_ctx_ptr.get())->close(state);
+ _vconjunct_ctx_ptr = nullptr;
+ }
}
+ // filter idle conjunct in vexpr_contexts
+ _peel_pushed_conjuncts();
}
void OlapScanNode::eval_const_conjuncts() {
@@ -843,11 +849,6 @@ static bool ignore_cast(SlotDescriptor* slot, Expr* expr) {
bool OlapScanNode::should_push_down_in_predicate(doris::SlotDescriptor* slot,
doris::InPredicate* pred) {
- if (pred->is_not_in()) {
- // can not push down NOT IN predicate to storage engine
- return false;
- }
-
if (Expr::type_without_cast(pred->get_child(0)) != TExprNodeType::SLOT_REF) {
// not a slot ref(column)
return false;
@@ -1672,4 +1673,46 @@ Status OlapScanNode::add_one_batch(RowBatch* row_batch) {
}
+vectorized::VExpr* OlapScanNode::_dfs_peel_conjunct(vectorized::VExpr* expr, int& leaf_index) {
+ static constexpr auto is_leaf = [](vectorized::VExpr* expr) { return !expr->is_and_expr(); };
+
+ if (is_leaf(expr)) {
+ return _pushed_conjuncts_index.count(leaf_index++) ? nullptr : expr;
+ } else {
+ vectorized::VExpr* left_child = _dfs_peel_conjunct(expr->children()[0], leaf_index);
+ vectorized::VExpr* right_child = _dfs_peel_conjunct(expr->children()[1], leaf_index);
+
+ if (left_child != nullptr && right_child != nullptr) {
+ expr->set_children({left_child, right_child});
+ return expr;
+ }
+ // here do not close Expr* now
+ return left_child != nullptr ? left_child : right_child;
+ }
+}
+
+// This function is used to remove pushed expr in expr tree.
+// It relies on the logic of function convertConjunctsToAndCompoundPredicate() of FE splicing expr.
+// It requires FE to satisfy each splicing with 'and' expr, and spliced from left to right, in order.
+// Expr tree specific forms do not require requirements.
+void OlapScanNode::_peel_pushed_conjuncts() {
+ if (_vconjunct_ctx_ptr.get() == nullptr) return;
+
+ int leaf_index = 0;
+ vectorized::VExpr* conjunct_expr_root = (*_vconjunct_ctx_ptr.get())->root();
+
+ if (conjunct_expr_root != nullptr) {
+ vectorized::VExpr* new_conjunct_expr_root =
+ _dfs_peel_conjunct(conjunct_expr_root, leaf_index);
+ if (new_conjunct_expr_root == nullptr) {
+ _vconjunct_ctx_ptr = nullptr;
+ _scanner_profile->add_info_string("VconjunctExprTree", "null");
+ } else {
+ (*_vconjunct_ctx_ptr.get())->set_root(new_conjunct_expr_root);
+ _scanner_profile->add_info_string("VconjunctExprTree",
+ new_conjunct_expr_root->debug_string());
+ }
+ }
+}
+
} // namespace doris
diff --git a/be/src/exec/olap_scan_node.h b/be/src/exec/olap_scan_node.h
index 3b7df03..f9905e2 100644
--- a/be/src/exec/olap_scan_node.h
+++ b/be/src/exec/olap_scan_node.h
@@ -32,6 +32,8 @@
#include "runtime/vectorized_row_batch.h"
#include "util/progress_updater.h"
#include "util/spinlock.h"
+#include "vec/exec/volap_scanner.h"
+#include "vec/exprs/vexpr.h"
namespace doris {
class IRuntimeFilter;
@@ -158,6 +160,7 @@ protected:
RuntimeProfile* profile);
friend class OlapScanner;
+ friend class doris::vectorized::VOlapScanner;
// Tuple id resolved in prepare() to set _tuple_desc;
TupleId _tuple_id;
@@ -215,7 +218,7 @@ protected:
std::mutex _scan_batches_lock;
std::condition_variable _scan_batch_added_cv;
- int64_t _running_thread = 0;
+ std::atomic_int _running_thread = 0;
std::condition_variable _scan_thread_exit_cv;
std::list<RowBatch*> _scan_row_batches;
@@ -321,6 +324,9 @@ protected:
RuntimeProfile::Counter* _scanner_wait_worker_timer = nullptr;
RuntimeProfile::Counter* _olap_wait_batch_queue_timer = nullptr;
+
+ vectorized::VExpr* _dfs_peel_conjunct(vectorized::VExpr* expr, int& leaf_index);
+ void _peel_pushed_conjuncts(); // remove pushed expr from conjunct tree
};
} // namespace doris
diff --git a/be/src/exec/olap_scanner.cpp b/be/src/exec/olap_scanner.cpp
index af1d658..34336fa 100644
--- a/be/src/exec/olap_scanner.cpp
+++ b/be/src/exec/olap_scanner.cpp
@@ -19,9 +19,9 @@
#include <string>
-#include "gen_cpp/PaloInternalService_types.h"
#include "common/utils.h"
#include "exprs/expr_context.h"
+#include "gen_cpp/PaloInternalService_types.h"
#include "olap/decimal12.h"
#include "olap/field.h"
#include "olap/uint24.h"
@@ -176,8 +176,10 @@ Status OlapScanner::_init_tablet_reader_params(
_tablet_reader_params.rs_readers[1]->rowset()->start_version() == 2 &&
!_tablet_reader_params.rs_readers[1]->rowset()->rowset_meta()->is_segments_overlapping());
+ _params.origin_return_columns = &_return_columns;
if (_aggregation || single_version) {
_tablet_reader_params.return_columns = _return_columns;
+ _tablet_reader_params.direct_mode = true;
} else {
// we need to fetch all key columns to do the right aggregation on storage engine side.
for (size_t i = 0; i < _tablet->num_key_columns(); ++i) {
@@ -239,7 +241,8 @@ Status OlapScanner::_init_return_columns() {
}
}
if (auto sequence_col_idx = _tablet->tablet_schema().sequence_col_idx();
- has_replace_col && std::find(_return_columns.begin(), _return_columns.end(), sequence_col_idx) == _return_columns.end()) {
+ has_replace_col && std::find(_return_columns.begin(), _return_columns.end(),
+ sequence_col_idx) == _return_columns.end()) {
_return_columns.push_back(sequence_col_idx);
}
}
diff --git a/be/src/exec/partitioned_aggregation_node.cc b/be/src/exec/partitioned_aggregation_node.cc
index 2311f82..68f536f 100644
--- a/be/src/exec/partitioned_aggregation_node.cc
+++ b/be/src/exec/partitioned_aggregation_node.cc
@@ -1048,7 +1048,7 @@ Tuple* PartitionedAggregationNode::GetOutputTuple(const vector<NewAggFnEvaluator
}
if (needs_finalize_) {
NewAggFnEvaluator::Finalize(agg_fn_evals, tuple, dst,
- grouping_exprs_.size() == 0 && child(0)->rows_returned() == 0);
+ grouping_exprs_.size() == 0 && child(0)->rows_returned() == 0);
} else {
NewAggFnEvaluator::Serialize(agg_fn_evals, tuple);
}
diff --git a/be/src/exec/scan_node.h b/be/src/exec/scan_node.h
index 5f8ec4b..bd09991 100644
--- a/be/src/exec/scan_node.h
+++ b/be/src/exec/scan_node.h
@@ -91,7 +91,7 @@ public:
protected:
RuntimeProfile::Counter* _bytes_read_counter; // # bytes read from the scanner
- // # rows/tuples read from the scanner (including those discarded by eval_conjucts())
+ // # rows/tuples read from the scanner (including those discarded by eval_conjuncts())
RuntimeProfile::Counter* _rows_read_counter;
// Wall based aggregate read throughput [bytes/sec]
RuntimeProfile::Counter* _total_throughput_counter;
diff --git a/be/src/exec/schema_scan_node.h b/be/src/exec/schema_scan_node.h
index 2052de0..f45086c 100644
--- a/be/src/exec/schema_scan_node.h
+++ b/be/src/exec/schema_scan_node.h
@@ -62,7 +62,7 @@ private:
void debug_string(int indentation_level, std::stringstream* out) const override;
// Copy one row from schema table to input tuple
void copy_one_row();
-
+protected:
bool _is_init;
const std::string _table_name;
SchemaScannerParam _scanner_param;
diff --git a/be/src/exec/tablet_info.cpp b/be/src/exec/tablet_info.cpp
index 7d470bf..a1195d5 100644
--- a/be/src/exec/tablet_info.cpp
+++ b/be/src/exec/tablet_info.cpp
@@ -407,4 +407,228 @@ uint32_t OlapTablePartitionParam::_compute_dist_hash(Tuple* key) const {
return hash_val;
}
+VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<OlapTableSchemaParam>& schema,
+ const TOlapTablePartitionParam& t_param)
+ : _schema(schema),
+ _t_param(t_param),
+ _slots(_schema->tuple_desc()->slots()),
+ _mem_tracker(MemTracker::CreateTracker(-1, "OlapTablePartitionParam")) {
+ for (auto slot : _slots) {
+ _partition_block.insert({slot->get_empty_mutable_column(), slot->get_data_type_ptr(), slot->col_name()});
+ }
+}
+
+VOlapTablePartitionParam::~VOlapTablePartitionParam() {
+ _mem_tracker->Release(_mem_usage);
+}
+
+Status VOlapTablePartitionParam::init() {
+ std::vector<std::string> slot_column_names;
+ for (auto slot_desc : _schema->tuple_desc()->slots()) {
+ slot_column_names.emplace_back(slot_desc->col_name());
+ }
+
+ auto find_slot_locs = [&slot_column_names](const std::string& slot_name, std::vector<uint16_t>& locs, const std::string& column_type) {
+ auto it = std::find(slot_column_names.begin(), slot_column_names.end(), slot_name);
+ if (it == slot_column_names.end()) {
+ return Status::InternalError(column_type + " column not found, column =" + slot_name);
+ }
+ locs.emplace_back(it - slot_column_names.begin());
+ return Status::OK();
+ };
+
+ if (_t_param.__isset.partition_columns) {
+ for (auto& part_col : _t_param.partition_columns) {
+ RETURN_IF_ERROR(find_slot_locs(part_col, _partition_slot_locs, "partition"));
+ }
+ }
+
+ _partitions_map.reset(new std::map<BlockRow*, VOlapTablePartition*, VOlapTablePartKeyComparator>(
+ VOlapTablePartKeyComparator(_partition_slot_locs)));
+ if (_t_param.__isset.distributed_columns) {
+ for (auto& col : _t_param.distributed_columns) {
+ RETURN_IF_ERROR(find_slot_locs(col, _distributed_slot_locs, "distributed"));
+ }
+ }
+
+ DCHECK(!_t_param.partitions.empty()) << "must have at least 1 partition";
+ _is_in_partition = _t_param.partitions[0].__isset.in_keys;
+
+ // initial partitions
+ for (int i = 0; i < _t_param.partitions.size(); ++i) {
+ const TOlapTablePartition& t_part = _t_param.partitions[i];
+ auto part = _obj_pool.add(new VOlapTablePartition(&_partition_block));
+ part->id = t_part.id;
+
+ if (!_is_in_partition) {
+ if (t_part.__isset.start_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(t_part.start_keys, &part->start_key));
+ }
+
+ if (t_part.__isset.end_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(t_part.end_keys, &part->end_key));
+ }
+ } else {
+ for (const auto& keys : t_part.in_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(keys, &part->in_keys.emplace_back(&_partition_block, -1)));
+ }
+ }
+
+ part->num_buckets = t_part.num_buckets;
+ auto num_indexes = _schema->indexes().size();
+ if (t_part.indexes.size() != num_indexes) {
+ std::stringstream ss;
+ ss << "number of partition's index is not equal with schema's"
+ << ", num_part_indexes=" << t_part.indexes.size()
+ << ", num_schema_indexes=" << num_indexes;
+ return Status::InternalError(ss.str());
+ }
+ part->indexes = t_part.indexes;
+ std::sort(part->indexes.begin(), part->indexes.end(),
+ [](const OlapTableIndexTablets& lhs, const OlapTableIndexTablets& rhs) {
+ return lhs.index_id < rhs.index_id;
+ });
+ // check index
+ for (int j = 0; j < num_indexes; ++j) {
+ if (part->indexes[j].index_id != _schema->indexes()[j]->index_id) {
+ std::stringstream ss;
+ ss << "partition's index is not equal with schema's"
+ << ", part_index=" << part->indexes[j].index_id
+ << ", schema_index=" << _schema->indexes()[j]->index_id;
+ return Status::InternalError(ss.str());
+ }
+ }
+ _partitions.emplace_back(part);
+ if (_is_in_partition) {
+ for (auto& in_key : part->in_keys) {
+ _partitions_map->emplace(&in_key, part);
+ }
+ } else {
+ _partitions_map->emplace(&part->end_key, part);
+ }
+ }
+
+ _mem_usage = _partition_block.allocated_bytes();
+ _mem_tracker->Consume(_mem_usage);
+ return Status::OK();
+}
+
+bool VOlapTablePartitionParam::find_tablet(BlockRow* block_row, const VOlapTablePartition** partition,
+ uint32_t* dist_hashes) const {
+ auto it = _is_in_partition ? _partitions_map->find(block_row) : _partitions_map->upper_bound(block_row);
+ if (it == _partitions_map->end()) {
+ return false;
+ }
+ if (_is_in_partition || _part_contains(it->second, block_row)) {
+ *partition = it->second;
+ *dist_hashes = _compute_dist_hash(block_row);
+ return true;
+ }
+ return false;
+}
+
+Status VOlapTablePartitionParam::_create_partition_keys(const std::vector<TExprNode>& t_exprs,
+ BlockRow* part_key) {
+ for (int i = 0; i < t_exprs.size(); i++) {
+ RETURN_IF_ERROR(_create_partition_key(t_exprs[i], part_key->first,
+ _partition_slot_locs[i]));
+ }
+ return Status::OK();
+}
+
+Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr, vectorized::Block* block,
+ uint16_t pos) {
+ auto column = std::move(*block->get_by_position(pos).column).mutate();
+ switch (t_expr.node_type) {
+ case TExprNodeType::DATE_LITERAL: {
+ vectorized::VecDateTimeValue dt;
+ if (!dt.from_date_str(
+ t_expr.date_literal.value.c_str(), t_expr.date_literal.value.size())) {
+ std::stringstream ss;
+ ss << "invalid date literal in partition column, date=" << t_expr.date_literal;
+ return Status::InternalError(ss.str());
+ }
+ column->insert_data(reinterpret_cast<const char *>(&dt), 0);
+ break;
+ }
+ case TExprNodeType::INT_LITERAL: {
+ switch (t_expr.type.types[0].scalar_type.type) {
+ case TPrimitiveType::TINYINT: {
+ int8_t value = t_expr.int_literal.value;
+ column->insert_data(reinterpret_cast<const char *>(&value), 0);
+ break;
+ }
+ case TPrimitiveType::SMALLINT: {
+ int16_t value = t_expr.int_literal.value;
+ column->insert_data(reinterpret_cast<const char *>(&value), 0);
+ break;
+ }
+ case TPrimitiveType::INT: {
+ int32_t value = t_expr.int_literal.value;
+ column->insert_data(reinterpret_cast<const char *>(&value), 0);
+ break;
+ }
+ default:
+ int64_t value = t_expr.int_literal.value;
+ column->insert_data(reinterpret_cast<const char *>(&value), 0);
+ }
+ break;
+ } case TExprNodeType::LARGE_INT_LITERAL: {
+ StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+ __int128 value = StringParser::string_to_int<__int128>(t_expr.large_int_literal.value.c_str(),
+ t_expr.large_int_literal.value.size(),
+ &parse_result);
+ if (parse_result != StringParser::PARSE_SUCCESS) {
+ value = MAX_INT128;
+ }
+ column->insert_data(reinterpret_cast<const char *>(&value), 0);
+ break;
+ } case TExprNodeType::STRING_LITERAL: {
+ int len = t_expr.string_literal.value.size();
+ const char* str_val = t_expr.string_literal.value.c_str();
+
+ // CHAR is a fixed-length string and needs to use the length in the slot definition,
+ // VARVHAR is a variable-length string and needs to use the length of the string itself
+ // padding 0 to CHAR field
+// if (TYPE_CHAR == slot_desc->type().type && len < slot_desc->type().len) {
+// auto new_ptr = (char*)_mem_pool->allocate(slot_desc->type().len);
+// memset(new_ptr, 0, slot_desc->type().len);
+// memcpy(new_ptr, str_val, len);
+//
+// str_val = new_ptr;
+// len = slot_desc->type().len;
+// }
+ column->insert_data(str_val, len);
+ break;
+ } case TExprNodeType::BOOL_LITERAL: {
+ column->insert_data(reinterpret_cast<const char *>(&t_expr.bool_literal.value), 0);
+ break;
+ } default: {
+ std::stringstream ss;
+ ss << "unsupported partition column node type, type=" << t_expr.node_type;
+ return Status::InternalError(ss.str());
+ }
+ }
+ return Status::OK();
+}
+
+uint32_t VOlapTablePartitionParam::_compute_dist_hash(BlockRow* key) const {
+ uint32_t hash_val = 0;
+ for (int i = 0; i < _distributed_slot_locs.size(); ++i) {
+ auto slot_desc = _slots[_distributed_slot_locs[i]];
+ auto column = key->first->get_by_position(_distributed_slot_locs[i]).column;
+
+ auto val = column->get_data_at(key->second);
+ if (val.data != nullptr) {
+ hash_val = RawValue::zlib_crc32(val.data, val.size, slot_desc->type().type, hash_val);
+ } else {
+ // NULL is treat as 0 when hash
+ static const int INT_VALUE = 0;
+ static const TypeDescriptor INT_TYPE(TYPE_INT);
+ hash_val = RawValue::zlib_crc32(&INT_VALUE, INT_TYPE, hash_val);
+ }
+ }
+ return hash_val;
+}
+
} // namespace doris
diff --git a/be/src/exec/tablet_info.h b/be/src/exec/tablet_info.h
index 7bb2640..53221ba 100644
--- a/be/src/exec/tablet_info.h
+++ b/be/src/exec/tablet_info.h
@@ -31,6 +31,8 @@
#include "runtime/raw_value.h"
#include "runtime/tuple.h"
+#include "vec/core/block.h"
+
namespace doris {
class MemPool;
@@ -205,6 +207,104 @@ private:
_partitions_map;
};
+using BlockRow = std::pair<vectorized::Block*, int32_t>;
+
+struct VOlapTablePartition {
+ int64_t id = 0;
+ BlockRow start_key;
+ BlockRow end_key;
+ std::vector<BlockRow> in_keys;
+ int64_t num_buckets = 0;
+ std::vector<OlapTableIndexTablets> indexes;
+
+ VOlapTablePartition(vectorized::Block* partition_block):
+ start_key{partition_block, -1}, end_key{partition_block, -1} {};
+};
+
+class VOlapTablePartKeyComparator {
+public:
+ VOlapTablePartKeyComparator(const std::vector<uint16_t>& slot_locs)
+ : _slot_locs(slot_locs) {}
+
+ // return true if lhs < rhs
+ // 'row' is -1 mean
+ bool operator()(const BlockRow* lhs, const BlockRow* rhs) const {
+ if (lhs->second == -1) {
+ return false;
+ } else if (rhs->second == -1) {
+ return true;
+ }
+
+ for (auto slot_loc : _slot_locs) {
+ auto res = lhs->first->get_by_position(slot_loc).column->compare_at(
+ lhs->second, rhs->second, *rhs->first->get_by_position(slot_loc).column, -1);
+ if (res != 0) {
+ return res < 0;
+ }
+ }
+ // equal, return false
+ return false;
+ }
+
+private:
+ const std::vector<uint16_t>& _slot_locs;
+};
+
+// store an olap table's tablet information
+class VOlapTablePartitionParam {
+public:
+ VOlapTablePartitionParam(std::shared_ptr<OlapTableSchemaParam>& schema,
+ const TOlapTablePartitionParam& param);
+
+ ~VOlapTablePartitionParam();
+
+ Status init();
+
+ int64_t db_id() const { return _t_param.db_id; }
+ int64_t table_id() const { return _t_param.table_id; }
+ int64_t version() const { return _t_param.version; }
+
+ // return true if we found this tuple in partition
+ bool find_tablet(BlockRow* block_row, const VOlapTablePartition** partitions,
+ uint32_t* dist_hash) const;
+
+ const std::vector<VOlapTablePartition*>& get_partitions() const { return _partitions; }
+
+private:
+ Status _create_partition_keys(const std::vector<TExprNode>& t_exprs, BlockRow* part_key);
+
+ Status _create_partition_key(const TExprNode& t_expr, vectorized::Block* block, uint16_t pos);
+
+ uint32_t _compute_dist_hash(BlockRow* key) const;
+
+ // check if this partition contain this key
+ bool _part_contains(VOlapTablePartition* part, BlockRow* key) const {
+ // start_key.second == -1 means only single partition
+ VOlapTablePartKeyComparator comparator(_partition_slot_locs);
+ return part->start_key.second == -1 || !comparator(key, &part->start_key);
+ }
+
+private:
+ // this partition only valid in this schema
+ std::shared_ptr<OlapTableSchemaParam> _schema;
+ TOlapTablePartitionParam _t_param;
+
+ const std::vector<SlotDescriptor*>& _slots;
+ std::vector<uint16_t> _partition_slot_locs;
+ std::vector<uint16_t> _distributed_slot_locs;
+
+ ObjectPool _obj_pool;
+ vectorized::Block _partition_block;
+ std::shared_ptr<MemTracker> _mem_tracker;
+ std::vector<VOlapTablePartition*> _partitions;
+ std::unique_ptr<std::map<BlockRow*, VOlapTablePartition*, VOlapTablePartKeyComparator>>
+ _partitions_map;
+
+ bool _is_in_partition = false;
+ uint32_t _mem_usage = 0;
+};
+
+
using TabletLocation = TTabletLocation;
// struct TTabletLocation {
// 1: required i64 tablet_id
diff --git a/be/src/exec/tablet_sink.cpp b/be/src/exec/tablet_sink.cpp
index 1718f7d..7acde28 100644
--- a/be/src/exec/tablet_sink.cpp
+++ b/be/src/exec/tablet_sink.cpp
@@ -251,6 +251,53 @@ Status NodeChannel::add_row(Tuple* input_tuple, int64_t tablet_id) {
return Status::OK();
}
+Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
+ // If add_row() when _eos_is_produced==true, there must be sth wrong, we can only mark this channel as failed.
+ auto st = none_of({_cancelled, _eos_is_produced});
+ if (!st.ok()) {
+ if (_cancelled) {
+ std::lock_guard<SpinLock> l(_cancel_msg_lock);
+ return Status::InternalError("add row failed. " + _cancel_msg);
+ } else {
+ return st.clone_and_prepend("already stopped, can't add row. cancelled/eos: ");
+ }
+ }
+
+ // We use OlapTableSink mem_tracker which has the same ancestor of _plan node,
+ // so in the ideal case, mem limit is a matter for _plan node.
+ // But there is still some unfinished things, we do mem limit here temporarily.
+ // _cancelled may be set by rpc callback, and it's possible that _cancelled might be set in any of the steps below.
+ // It's fine to do a fake add_row() and return OK, because we will check _cancelled in next add_row() or mark_close().
+ while (!_cancelled && _parent->_mem_tracker->AnyLimitExceeded(MemLimit::HARD) &&
+ _pending_batches_num > 0) {
+ SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
+ SleepFor(MonoDelta::FromMilliseconds(10));
+ }
+
+ auto row_no = _cur_batch->add_row();
+ if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ {
+ SCOPED_ATOMIC_TIMER(&_queue_push_lock_ns);
+ std::lock_guard<std::mutex> l(_pending_batches_lock);
+ //To simplify the add_row logic, postpone adding batch into req until the time of sending req
+ _pending_batches.emplace(std::move(_cur_batch), _cur_add_batch_request);
+ _pending_batches_num++;
+ }
+
+ _cur_batch.reset(new RowBatch(*_row_desc, _batch_size, _parent->_mem_tracker.get()));
+ _cur_add_batch_request.clear_tablet_ids();
+
+ row_no = _cur_batch->add_row();
+ }
+ DCHECK_NE(row_no, RowBatch::INVALID_ROW_INDEX);
+
+ _cur_batch->get_row(row_no)->set_tuple(0,
+ block_row.first->deep_copy_tuple(*_tuple_desc, _cur_batch->tuple_data_pool(), block_row.second, 0, true));
+ _cur_batch->commit_last_row();
+ _cur_add_batch_request.add_tablet_ids(tablet_id);
+ return Status::OK();
+}
+
Status NodeChannel::mark_close() {
auto st = none_of({_cancelled, _eos_is_produced});
if (!st.ok()) {
@@ -501,6 +548,29 @@ Status IndexChannel::add_row(Tuple* tuple, int64_t tablet_id) {
return Status::OK();
}
+Status IndexChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
+ auto it = _channels_by_tablet.find(tablet_id);
+ DCHECK(it != _channels_by_tablet.end()) << "unknown tablet, tablet_id=" << tablet_id;
+ std::stringstream ss;
+ for (auto channel : it->second) {
+ // if this node channel is already failed, this add_row will be skipped
+ auto st = channel->add_row(block_row, tablet_id);
+ if (!st.ok()) {
+ mark_as_failed(channel);
+ ss << st.get_error_msg() << "; ";
+ }
+ }
+
+ if (has_intolerable_failure()) {
+ std::stringstream ss2;
+ ss2 << "index channel has intolerable failure. " << BackendOptions::get_localhost()
+ << ", err: " << ss.str();
+ return Status::InternalError(ss2.str());
+ }
+
+ return Status::OK();
+}
+
bool IndexChannel::has_intolerable_failure() {
for (const auto& it : _failed_channels) {
if (it.second.size() >= ((_parent->_num_replicas + 1) / 2)) {
@@ -731,7 +801,7 @@ Status OlapTableSink::send(RuntimeState* state, RowBatch* input_batch) {
batch = _output_batch.get();
}
int num_invalid_rows = 0;
- if (_need_validate_data) {
+ {
SCOPED_RAW_TIMER(&_validate_data_ns);
_filter_bitmap.Reset(batch->num_rows());
num_invalid_rows = _validate_data(state, batch, &_filter_bitmap);
diff --git a/be/src/exec/tablet_sink.h b/be/src/exec/tablet_sink.h
index b5e86f7..6f403b1 100644
--- a/be/src/exec/tablet_sink.h
+++ b/be/src/exec/tablet_sink.h
@@ -160,6 +160,8 @@ public:
Status add_row(Tuple* tuple, int64_t tablet_id);
+ Status add_row(BlockRow& block_row, int64_t tablet_id);
+
// two ways to stop channel:
// 1. mark_close()->close_wait() PS. close_wait() will block waiting for the last AddBatch rpc response.
// 2. just cancel()
@@ -276,6 +278,8 @@ public:
Status add_row(Tuple* tuple, int64_t tablet_id);
+ Status add_row(BlockRow& block_row, int64_t tablet_id);
+
void for_each_node_channel(const std::function<void(NodeChannel*)>& func) {
for (auto& it : _node_channels) {
func(it.second);
@@ -350,7 +354,7 @@ private:
// only focus on pending batches and channel status, the internal errors of NodeChannels will be handled by the producer
void _send_batch_process();
-private:
+protected:
friend class NodeChannel;
friend class IndexChannel;
diff --git a/be/src/exec/text_converter.cpp b/be/src/exec/text_converter.cpp
index 7801f66..e2d37d5 100644
--- a/be/src/exec/text_converter.cpp
+++ b/be/src/exec/text_converter.cpp
@@ -59,4 +59,26 @@ void TextConverter::unescape_string(const char* src, char* dest, size_t* len) {
*len = dest_ptr - dest_start;
}
+void TextConverter::unescape_string_on_spot(const char* src, size_t* len) {
+ char* dest_ptr = const_cast<char*>(src);
+ const char* end = src + *len;
+ bool escape_next_char = false;
+
+ while (src < end) {
+ if (*src == _escape_char) {
+ escape_next_char = !escape_next_char;
+ } else {
+ escape_next_char = false;
+ }
+
+ if (escape_next_char) {
+ ++src;
+ } else {
+ *dest_ptr++ = *src++;
+ }
+ }
+
+ *len = dest_ptr - src;
+}
+
} // namespace doris
diff --git a/be/src/exec/text_converter.h b/be/src/exec/text_converter.h
index 4d97b7c..36b3254 100644
--- a/be/src/exec/text_converter.h
+++ b/be/src/exec/text_converter.h
@@ -19,7 +19,7 @@
#define DORIS_BE_SRC_QUERY_EXEC_TEXT_CONVERTER_H
#include "runtime/runtime_state.h"
-
+#include "vec/core/block.h"
namespace doris {
class MemPool;
@@ -48,11 +48,14 @@ public:
bool write_slot(const SlotDescriptor* slot_desc, Tuple* tuple, const char* data, int len,
bool copy_string, bool need_escape, MemPool* pool);
+ bool write_column(const SlotDescriptor* slot_desc, vectorized::MutableColumnPtr* column_ptr,
+ const char* data, size_t len, bool copy_string, bool need_escape);
+
// Removes escape characters from len characters of the null-terminated string src,
// and copies the unescaped string into dest, changing *len to the unescaped length.
// No null-terminator is added to dest.
void unescape_string(const char* src, char* dest, size_t* len);
-
+ void unescape_string_on_spot(const char* src, size_t* len);
// Removes escape characters from 'str', allocating a new string from pool.
// 'str' is updated with the new ptr and length.
void unescape_string(StringValue* str, MemPool* pool);
diff --git a/be/src/exec/text_converter.hpp b/be/src/exec/text_converter.hpp
index 1b1a60e..44e807c 100644
--- a/be/src/exec/text_converter.hpp
+++ b/be/src/exec/text_converter.hpp
@@ -17,6 +17,7 @@
#ifndef DORIS_BE_SRC_QUERY_EXEC_TEXT_CONVERTER_HPP
#define DORIS_BE_SRC_QUERY_EXEC_TEXT_CONVERTER_HPP
+#include <sql.h>
#include <boost/algorithm/string.hpp>
@@ -32,7 +33,7 @@
#include "util/binary_cast.hpp"
#include "util/string_parser.hpp"
#include "util/types.h"
-
+#include "vec/runtime/vdatetime_value.h"
namespace doris {
// Note: this function has a codegen'd version. Changing this function requires
@@ -165,6 +166,134 @@ inline bool TextConverter::write_slot(const SlotDescriptor* slot_desc, Tuple* tu
return true;
}
+inline bool TextConverter::write_column(const SlotDescriptor* slot_desc,
+ vectorized::MutableColumnPtr* column_ptr, const char* data,
+ size_t len, bool copy_string, bool need_escape) {
+ vectorized::IColumn* col_ptr = column_ptr->get();
+ // \N means it's NULL
+ if (true == slot_desc->is_nullable()) {
+ auto* nullable_column = reinterpret_cast<vectorized::ColumnNullable*>(column_ptr->get());
+ if ((len == 2 && data[0] == '\\' && data[1] == 'N') || len == SQL_NULL_DATA) {
+ nullable_column->insert_data(nullptr, 0);
+ return true;
+ } else {
+ nullable_column->get_null_map_data().push_back(0);
+ col_ptr = &nullable_column->get_nested_column();
+ }
+ }
+ StringParser::ParseResult parse_result = StringParser::PARSE_SUCCESS;
+
+ // Parse the raw-text data. Translate the text string to internal format.
+ switch (slot_desc->type().type) {
+ case TYPE_HLL:
+ case TYPE_VARCHAR:
+ case TYPE_CHAR: {
+ if (need_escape) {
+ unescape_string_on_spot(data, &len);
+ }
+ reinterpret_cast<vectorized::ColumnString*>(col_ptr)->insert_data(data, len);
+ break;
+ }
+
+ case TYPE_BOOLEAN: {
+ bool num = StringParser::string_to_bool(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::UInt8>*>(col_ptr)->insert_value(
+ (uint8_t)num);
+ break;
+ }
+ case TYPE_TINYINT: {
+ int8_t num = StringParser::string_to_int<int8_t>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int8>*>(col_ptr)->insert_value(num);
+ break;
+ }
+ case TYPE_SMALLINT: {
+ int16_t num = StringParser::string_to_int<int16_t>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int16>*>(col_ptr)->insert_value(num);
+ break;
+ }
+ case TYPE_INT: {
+ int32_t num = StringParser::string_to_int<int32_t>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int32>*>(col_ptr)->insert_value(num);
+ break;
+ }
+ case TYPE_BIGINT: {
+ int64_t num = StringParser::string_to_int<int64_t>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int64>*>(col_ptr)->insert_value(num);
+ break;
+ }
+ case TYPE_LARGEINT: {
+ __int128 num = StringParser::string_to_int<__int128>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int128>*>(col_ptr)->insert_value(num);
+ break;
+ }
+
+ case TYPE_FLOAT: {
+ float num = StringParser::string_to_float<float>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Float32>*>(col_ptr)->insert_value(
+ num);
+ break;
+ }
+ case TYPE_DOUBLE: {
+ double num = StringParser::string_to_float<double>(data, len, &parse_result);
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Float64>*>(col_ptr)->insert_value(
+ num);
+ break;
+ }
+ case TYPE_DATE: {
+ vectorized::VecDateTimeValue ts_slot;
+ if (!ts_slot.from_date_str(data, len)) {
+ parse_result = StringParser::PARSE_FAILURE;
+ break;
+ }
+ ts_slot.cast_to_date();
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int64>*>(col_ptr)->insert_data(
+ reinterpret_cast<char*>(&ts_slot), 0);
+ break;
+ }
+
+ case TYPE_DATETIME: {
+ vectorized::VecDateTimeValue ts_slot;
+ if (!ts_slot.from_date_str(data, len)) {
+ parse_result = StringParser::PARSE_FAILURE;
+ break;
+ }
+ ts_slot.to_datetime();
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int64>*>(col_ptr)->insert_data(
+ reinterpret_cast<char*>(&ts_slot), 0);
+ break;
+ }
+
+ case TYPE_DECIMALV2: {
+ DecimalV2Value decimal_slot;
+ if (decimal_slot.parse_from_str(data, len)) {
+ parse_result = StringParser::PARSE_FAILURE;
+ break;
+ }
+ PackedInt128 num = binary_cast<DecimalV2Value, PackedInt128>(decimal_slot);
+ reinterpret_cast<vectorized::ColumnVector<doris::PackedInt128>*>(col_ptr)->insert_value(
+ num.value);
+ break;
+ }
+
+ default:
+ DCHECK(false) << "bad slot type: " << slot_desc->type();
+ break;
+ }
+
+ if (parse_result == StringParser::PARSE_FAILURE) {
+ if (true == slot_desc->is_nullable()) {
+ auto* nullable_column =
+ reinterpret_cast<vectorized::ColumnNullable*>(column_ptr->get());
+ size_t size = nullable_column->get_null_map_data().size();
+ doris::vectorized::NullMap& null_map_data = nullable_column->get_null_map_data();
+ null_map_data[size - 1] = 1;
+ } else {
+ return false;
+ }
+ }
+ return true;
+}
+
} // namespace doris
-#endif
+#endif
\ No newline at end of file
diff --git a/be/src/exprs/CMakeLists.txt b/be/src/exprs/CMakeLists.txt
index 5ebf1d1..5d69aa7 100644
--- a/be/src/exprs/CMakeLists.txt
+++ b/be/src/exprs/CMakeLists.txt
@@ -71,7 +71,6 @@ add_library(Exprs
hll_function.cpp
grouping_sets_functions.cpp
topn_function.cpp
-
table_function/explode_split.cpp
table_function/explode_bitmap.cpp
table_function/explode_json_array.cpp
diff --git a/be/src/exprs/aggregate_functions.cpp b/be/src/exprs/aggregate_functions.cpp
index 1f8db84..a22c3ee 100644
--- a/be/src/exprs/aggregate_functions.cpp
+++ b/be/src/exprs/aggregate_functions.cpp
@@ -592,30 +592,14 @@ void AggregateFunctions::sum(FunctionContext* ctx, const DecimalV2Val& src, Deci
}
if (dst->is_null) {
- dst->is_null = false;
- dst->set_to_zero();
+ init_zero_not_null<DecimalV2Val>(ctx, dst);
}
-
DecimalV2Value new_src = DecimalV2Value::from_decimal_val(src);
DecimalV2Value new_dst = DecimalV2Value::from_decimal_val(*dst);
new_dst = new_dst + new_src;
new_dst.to_decimal_val(dst);
}
-template <>
-void AggregateFunctions::sum(FunctionContext* ctx, const LargeIntVal& src, LargeIntVal* dst) {
- if (src.is_null) {
- return;
- }
-
- if (dst->is_null) {
- dst->is_null = false;
- dst->val = 0;
- }
-
- dst->val += src.val;
-}
-
template <typename T>
void AggregateFunctions::min_init(FunctionContext* ctx, T* dst) {
auto val = AnyValUtil::max_val<T>(ctx);
@@ -2395,6 +2379,8 @@ template void AggregateFunctions::sum<IntVal, BigIntVal>(FunctionContext*, const
BigIntVal* dst);
template void AggregateFunctions::sum<BigIntVal, BigIntVal>(FunctionContext*, const BigIntVal& src,
BigIntVal* dst);
+template void AggregateFunctions::sum<LargeIntVal, LargeIntVal>(FunctionContext*, const LargeIntVal& src,
+ LargeIntVal* dst);
template void AggregateFunctions::sum<FloatVal, DoubleVal>(FunctionContext*, const FloatVal& src,
DoubleVal* dst);
template void AggregateFunctions::sum<DoubleVal, DoubleVal>(FunctionContext*, const DoubleVal& src,
diff --git a/be/src/exprs/math_functions.cpp b/be/src/exprs/math_functions.cpp
index 787887d..00cf3bd 100644
--- a/be/src/exprs/math_functions.cpp
+++ b/be/src/exprs/math_functions.cpp
@@ -178,11 +178,11 @@ ONE_ARG_MATH_FN(ln, DoubleVal, DoubleVal, std::log);
ONE_ARG_MATH_FN(log10, DoubleVal, DoubleVal, std::log10);
ONE_ARG_MATH_FN(exp, DoubleVal, DoubleVal, std::exp);
-FloatVal MathFunctions::sign(FunctionContext* ctx, const DoubleVal& v) {
+TinyIntVal MathFunctions::sign(FunctionContext* ctx, const DoubleVal& v) {
if (v.is_null) {
- return FloatVal::null();
+ return TinyIntVal::null();
}
- return FloatVal((v.val > 0) ? 1.0f : ((v.val < 0) ? -1.0f : 0.0f));
+ return TinyIntVal((v.val > 0) ? 1 : ((v.val < 0) ? -1 : 0));
}
DoubleVal MathFunctions::radians(FunctionContext* ctx, const DoubleVal& v) {
diff --git a/be/src/exprs/math_functions.h b/be/src/exprs/math_functions.h
index 5d1258e..15d8749 100644
--- a/be/src/exprs/math_functions.h
+++ b/be/src/exprs/math_functions.h
@@ -50,7 +50,7 @@ public:
static doris_udf::IntVal abs(doris_udf::FunctionContext*, const doris_udf::SmallIntVal&);
static doris_udf::SmallIntVal abs(doris_udf::FunctionContext*, const doris_udf::TinyIntVal&);
- static doris_udf::FloatVal sign(doris_udf::FunctionContext* ctx, const doris_udf::DoubleVal& v);
+ static doris_udf::TinyIntVal sign(doris_udf::FunctionContext* ctx, const doris_udf::DoubleVal& v);
static doris_udf::DoubleVal sin(doris_udf::FunctionContext*, const doris_udf::DoubleVal&);
static doris_udf::DoubleVal asin(doris_udf::FunctionContext*, const doris_udf::DoubleVal&);
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index b77cf9d..3cf45c4 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -373,6 +373,35 @@ public:
break;
}
}
+ void insert(const StringRef& value) {
+ switch (_column_return_type) {
+ case TYPE_DATE:
+ case TYPE_DATETIME: {
+ // DateTime->DateTimeValue
+ vectorized::DateTime date_time =*reinterpret_cast<const vectorized::DateTime*>(value.data);
+ vectorized::VecDateTimeValue vec_date_time_value = binary_cast<vectorized::Int64, vectorized::VecDateTimeValue>(date_time);
+ doris::DateTimeValue date_time_value;
+ vec_date_time_value.convert_vec_dt_to_dt(&date_time_value);
+ insert(reinterpret_cast<const void*>(&date_time_value));
+ break;
+ }
+
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_OBJECT:
+ case TYPE_STRING: {
+ // StringRef->StringValue
+ StringValue data = StringValue(const_cast<char*>(value.data), value.size);
+ insert(reinterpret_cast<const void*>(&data));
+ break;
+ }
+
+ default:
+ insert(reinterpret_cast<const void*>(value.data));
+ break;
+ }
+ }
template <class T>
Status get_push_context(T* container, RuntimeState* state, ExprContext* prob_expr) {
@@ -784,6 +813,11 @@ void IRuntimeFilter::insert(const void* data) {
}
}
+void IRuntimeFilter::insert(const StringRef& value) {
+ DCHECK(is_producer());
+ _wrapper->insert(value);
+}
+
Status IRuntimeFilter::publish() {
DCHECK(is_producer());
if (_has_local_target) {
diff --git a/be/src/exprs/runtime_filter.h b/be/src/exprs/runtime_filter.h
index 7d406f3..c5e597a 100644
--- a/be/src/exprs/runtime_filter.h
+++ b/be/src/exprs/runtime_filter.h
@@ -128,6 +128,7 @@ public:
// insert data to build filter
// only used for producer
void insert(const void* data);
+ void insert(const StringRef& data);
// publish filter
// push filter to remote node or push down it to scan_node
diff --git a/be/src/exprs/runtime_filter_slots.h b/be/src/exprs/runtime_filter_slots.h
index 576cad0..e098adc 100644
--- a/be/src/exprs/runtime_filter_slots.h
+++ b/be/src/exprs/runtime_filter_slots.h
@@ -20,9 +20,9 @@
#include "exprs/runtime_filter.h"
#include "runtime/runtime_filter_mgr.h"
#include "runtime/runtime_state.h"
+#include "vec/exprs/vexpr.h"
namespace doris {
-
// this class used in a hash join node
// Provide a unified interface for other classes
template <typename ExprCtxType>
@@ -129,6 +129,40 @@ public:
}
}
}
+ void insert(std::unordered_map<const vectorized::Block*, std::vector<int>>& datas) {
+ for (int i = 0; i < _build_expr_context.size(); ++i) {
+ auto iter = _runtime_filters.find(i);
+ if (iter == _runtime_filters.end()) continue;
+
+ int result_column_id = _build_expr_context[i]->get_last_result_column_id();
+ for (auto it : datas) {
+ auto& column = it.first->get_by_position(result_column_id).column;
+
+ if (auto* nullable =
+ vectorized::check_and_get_column<vectorized::ColumnNullable>(*column)) {
+ auto& column_nested = nullable->get_nested_column();
+ auto& column_nullmap = nullable->get_null_map_column();
+ for (int row_num : it.second) {
+ if (column_nullmap.get_bool(row_num)) {
+ continue;
+ }
+ const auto& ref_data = column_nested.get_data_at(row_num);
+ for (auto filter : iter->second) {
+ filter->insert(ref_data);
+ }
+ }
+
+ } else {
+ for (int row_num : it.second) {
+ const auto& ref_data = column->get_data_at(row_num);
+ for (auto filter : iter->second) {
+ filter->insert(ref_data);
+ }
+ }
+ }
+ }
+ }
+ }
// should call this method after insert
void ready_for_publish() {
@@ -166,5 +200,5 @@ private:
};
using RuntimeFilterSlots = RuntimeFilterSlotsBase<ExprContext>;
-
+using VRuntimeFilterSlots = RuntimeFilterSlotsBase<vectorized::VExprContext>;
} // namespace doris
diff --git a/be/src/exprs/v_string_functions.h b/be/src/exprs/v_string_functions.h
new file mode 100644
index 0000000..3fd9845
--- /dev/null
+++ b/be/src/exprs/v_string_functions.h
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#ifndef BE_V_STRING_FUNCTIONS_H
+#define BE_V_STRING_FUNCTIONS_H
+
+#include <stdint.h>
+#include <unistd.h>
+#include "runtime/string_value.hpp"
+
+#ifdef __SSE2__
+#include <emmintrin.h>
+#endif
+
+namespace doris {
+class VStringFunctions {
+public:
+#ifdef __SSE2__
+ /// n equals to 16 chars length
+ static constexpr auto REGISTER_SIZE = sizeof(__m128i);
+#endif
+public:
+ static StringVal rtrim(const StringVal& str) {
+ if (str.is_null || str.len == 0) {
+ return str;
+ }
+ auto begin = 0;
+ auto end = str.len - 1;
+#ifdef __SSE2__
+ char blank = ' ';
+ const auto pattern = _mm_set1_epi8(blank);
+ while (end - begin + 1 >= REGISTER_SIZE) {
+ const auto v_haystack = _mm_loadu_si128(reinterpret_cast<const __m128i *>(str.ptr + end + 1 - REGISTER_SIZE));
+ const auto v_against_pattern = _mm_cmpeq_epi8(v_haystack, pattern);
+ const auto mask = _mm_movemask_epi8(v_against_pattern);
+ int offset = __builtin_clz(~(mask << REGISTER_SIZE));
+ /// means not found
+ if (offset == 0)
+ {
+ return StringVal(str.ptr + begin, end - begin + 1);
+ } else {
+ end -= offset;
+ }
+ }
+#endif
+ while (end >= begin && str.ptr[end] == ' ') {
+ --end;
+ }
+ if (end < 0) {
+ return StringVal("");
+ }
+ return StringVal(str.ptr + begin, end - begin + 1);
+ }
+
+ static StringVal ltrim(const StringVal& str) {
+ if (str.is_null || str.len == 0) {
+ return str;
+ }
+ auto begin = 0;
+ auto end = str.len - 1;
+#ifdef __SSE2__
+ char blank = ' ';
+ const auto pattern = _mm_set1_epi8(blank);
+ while (end - begin + 1 >= REGISTER_SIZE) {
+ const auto v_haystack = _mm_loadu_si128(reinterpret_cast<const __m128i *>(str.ptr + begin));
+ const auto v_against_pattern = _mm_cmpeq_epi8(v_haystack, pattern);
+ const auto mask = _mm_movemask_epi8(v_against_pattern);
+ const auto offset = __builtin_ctz(mask ^ 0xffff);
+ /// means not found
+ if (offset == 0)
+ {
+ return StringVal(str.ptr + begin, end - begin + 1);
+ } else if (offset > REGISTER_SIZE) {
+ begin += REGISTER_SIZE;
+ } else {
+ begin += offset;
+ return StringVal(str.ptr + begin, end - begin + 1);
+ }
+ }
+#endif
+ while (begin <= end && str.ptr[begin] == ' ') {
+ ++begin;
+ }
+ return StringVal(str.ptr + begin, end - begin + 1);
+ }
+
+ static StringVal trim(const StringVal& str) {
+ if (str.is_null || str.len == 0) {
+ return str;
+ }
+ return rtrim(ltrim(str));
+ }
+
+ static bool is_ascii(StringVal str) {
+ #ifdef __SSE2__
+ size_t i = 0;
+ __m128i binary_code = _mm_setzero_si128();
+ if (str.len >= REGISTER_SIZE) {
+ for (; i <= str.len - REGISTER_SIZE; i += REGISTER_SIZE) {
+ __m128i chars = _mm_loadu_si128((const __m128i*)(str.ptr + i));
+ binary_code = _mm_or_si128(binary_code, chars);
+ }
+ }
+ int mask = _mm_movemask_epi8(binary_code);
+
+ char or_code = 0;
+ for (; i < str.len; i++) {
+ or_code |= str.ptr[i];
+ }
+ mask |= (or_code & 0x80);
+
+ return !mask;
+ #else
+ char or_code = 0;
+ for (size_t i = 0; i < str.len; i++) {
+ or_code |= str.ptr[i];
+ }
+ return !(or_code & 0x80);
+ #endif
+ }
+
+ static void reverse(const StringVal& str, StringVal dst) {
+ if (str.is_null) {
+ dst.ptr = NULL;
+ return;
+ }
+ const bool is_ascii = VStringFunctions::is_ascii(str);
+ if (is_ascii) {
+ int64_t begin = 0;
+ int64_t end = str.len;
+ int64_t result_end = dst.len;
+ #if defined(__SSE2__)
+ const auto shuffle_array = _mm_set_epi64((__m64)0x00'01'02'03'04'05'06'07ull, (__m64)0x08'09'0a'0b'0c'0d'0e'0full);
+ for (; (begin + REGISTER_SIZE) < end; begin += REGISTER_SIZE) {
+ result_end -= REGISTER_SIZE;
+ _mm_storeu_si128((__m128i*)(dst.ptr + result_end),
+ _mm_shuffle_epi8(_mm_loadu_si128((__m128i*)(str.ptr + begin)), shuffle_array));
+ }
+ #endif
+ for (; begin < end; ++begin) {
+ --result_end;
+ dst.ptr[result_end] = str.ptr[begin];
+ }
+ } else {
+ for (size_t i = 0, char_size = 0; i < str.len; i += char_size) {
+ char_size = get_utf8_byte_length((unsigned)(str.ptr)[i]);
+ std::copy(str.ptr + i, str.ptr + i + char_size, dst.ptr + str.len - i - char_size);
+ }
+ }
+ }
+
+ static size_t get_utf8_byte_length(unsigned char byte) {
+ size_t char_size = 0;
+ if (byte >= 0xFC) {
+ char_size = 6;
+ } else if (byte >= 0xF8) {
+ char_size = 5;
+ } else if (byte >= 0xF0) {
+ char_size = 4;
+ } else if (byte >= 0xE0) {
+ char_size = 3;
+ } else if (byte >= 0xC0) {
+ char_size = 2;
+ } else {
+ char_size = 1;
+ }
+ return char_size;
+ }
+
+ static void hex_encode(const unsigned char* src_str, size_t length, char* dst_str) {
+ static constexpr auto hex_table = "0123456789ABCDEF";
+ auto src_str_end = src_str + length;
+
+#if defined(__SSE2__)
+ constexpr auto step = sizeof(uint64);
+ if (src_str + step < src_str_end) {
+ const auto hex_map = _mm_loadu_si128(reinterpret_cast<const __m128i *>(hex_table));
+ const auto mask_map = _mm_set1_epi8(0x0F);
+
+ do {
+ auto data = _mm_loadu_si64(src_str);
+ auto hex_loc = _mm_and_si128(_mm_unpacklo_epi8(_mm_srli_epi64(data, 4), data), mask_map);
+ _mm_storeu_si128(reinterpret_cast<__m128i *>(dst_str), _mm_shuffle_epi8(hex_map, hex_loc));
+
+ src_str += step;
+ dst_str += step * 2;
+ } while (src_str + step < src_str_end);
+ }
+#endif
+ char res[2];
+ // hex(str) str length is n, result must be 2 * n length
+ for (; src_str < src_str_end; src_str += 1, dst_str += 2) {
+ // low 4 bits
+ *(res + 1) = hex_table[src_str[0] & 0x0F];
+ // high 4 bits
+ *res = hex_table[(src_str[0] >> 4)];
+ std::copy(res, res + 2, dst_str);
+ }
+ }
+};
+}
+
+#endif //BE_V_STRING_FUNCTIONS_H
\ No newline at end of file
diff --git a/be/src/olap/block_column_predicate.cpp b/be/src/olap/block_column_predicate.cpp
index f460be5..725ebda 100644
--- a/be/src/olap/block_column_predicate.cpp
+++ b/be/src/olap/block_column_predicate.cpp
@@ -39,6 +39,30 @@ void SingleColumnBlockPredicate::evaluate_or(RowBlockV2 *block, uint16_t selecte
_predicate->evaluate_or(&column_block, block->selection_vector(), selected_size, flags);
}
+void SingleColumnBlockPredicate::evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const {
+ auto column_id = _predicate->column_id();
+ auto& column = block[column_id];
+ _predicate->evaluate(*column, sel, selected_size);
+}
+
+void SingleColumnBlockPredicate::evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {
+ auto column_id = _predicate->column_id();
+ auto& column = block[column_id];
+ _predicate->evaluate_and(*column, sel, selected_size, flags);
+}
+
+void SingleColumnBlockPredicate::evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {
+ auto column_id = _predicate->column_id();
+ auto& column = block[column_id];
+ _predicate->evaluate_or(*column, sel, selected_size, flags);
+}
+
+void SingleColumnBlockPredicate::evaluate_vec(vectorized::MutableColumns& block, uint16_t size, bool* flags) const {
+ auto column_id = _predicate->column_id();
+ auto& column = block[column_id];
+ _predicate->evaluate_vec(*column, size, flags);
+}
+
void OrBlockColumnPredicate::evaluate(RowBlockV2* block, uint16_t* selected_size) const {
if (num_of_column_predicate() == 1) {
_block_column_predicate_vec[0]->evaluate(block, selected_size);
@@ -60,12 +84,39 @@ void OrBlockColumnPredicate::evaluate(RowBlockV2* block, uint16_t* selected_size
}
}
+void OrBlockColumnPredicate::evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const {
+ if (num_of_column_predicate() == 1) {
+ _block_column_predicate_vec[0]->evaluate(block, sel, selected_size);
+ } else {
+ bool ret_flags[*selected_size];
+ memset(ret_flags, false, *selected_size);
+ for (int i = 0; i < num_of_column_predicate(); ++i) {
+ auto column_predicate = _block_column_predicate_vec[i];
+ column_predicate->evaluate_or(block, sel, *selected_size, ret_flags);
+ }
+
+ uint16_t new_size = 0;
+ for (int i = 0; i < *selected_size; ++i) {
+ if (ret_flags[i]) {
+ sel[new_size++] = sel[i];
+ }
+ }
+ *selected_size = new_size;
+ }
+}
+
void OrBlockColumnPredicate::evaluate_or(RowBlockV2 *block, uint16_t selected_size, bool* flags) const {
for (auto block_column_predicate : _block_column_predicate_vec) {
block_column_predicate->evaluate_or(block, selected_size, flags);
}
}
+void OrBlockColumnPredicate::evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {
+ for (auto block_column_predicate : _block_column_predicate_vec) {
+ block_column_predicate->evaluate_or(block, sel, selected_size, flags);
+ }
+}
+
void OrBlockColumnPredicate::evaluate_and(RowBlockV2 *block, uint16_t selected_size, bool* flags) const {
if (num_of_column_predicate() == 1) {
_block_column_predicate_vec[0]->evaluate_and(block, selected_size, flags);
@@ -83,18 +134,47 @@ void OrBlockColumnPredicate::evaluate_and(RowBlockV2 *block, uint16_t selected_s
}
}
+void OrBlockColumnPredicate::evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {
+ if (num_of_column_predicate() == 1) {
+ _block_column_predicate_vec[0]->evaluate_and(block, sel, selected_size, flags);
+ } else {
+ bool ret_flags[selected_size];
+ memset(ret_flags, false, selected_size);
+ for (int i = 0; i < num_of_column_predicate(); ++i) {
+ auto column_predicate = _block_column_predicate_vec[i];
+ column_predicate->evaluate_or(block, sel, selected_size, ret_flags);
+ }
+
+ for (int i = 0; i < selected_size; ++i) {
+ flags[i] &= ret_flags[i];
+ }
+ }
+}
+
void AndBlockColumnPredicate::evaluate(RowBlockV2* block, uint16_t* selected_size) const {
for (auto block_column_predicate : _block_column_predicate_vec) {
block_column_predicate->evaluate(block, selected_size);
}
}
+void AndBlockColumnPredicate::evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const {
+ for (auto block_column_predicate : _block_column_predicate_vec) {
+ block_column_predicate->evaluate(block, sel, selected_size);
+ }
+}
+
void AndBlockColumnPredicate::evaluate_and(RowBlockV2 *block, uint16_t selected_size, bool* flags) const {
for (auto block_column_predicate : _block_column_predicate_vec) {
block_column_predicate->evaluate_and(block, selected_size, flags);
}
}
+void AndBlockColumnPredicate::evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {
+ for (auto block_column_predicate : _block_column_predicate_vec) {
+ block_column_predicate->evaluate_and(block, sel, selected_size, flags);
+ }
+}
+
void AndBlockColumnPredicate::evaluate_or(RowBlockV2 *block, uint16_t selected_size, bool* flags) const {
if (num_of_column_predicate() == 1) {
_block_column_predicate_vec[0]->evaluate_or(block, selected_size, flags);
@@ -113,4 +193,38 @@ void AndBlockColumnPredicate::evaluate_or(RowBlockV2 *block, uint16_t selected_s
}
}
+void AndBlockColumnPredicate::evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {
+ if (num_of_column_predicate() == 1) {
+ _block_column_predicate_vec[0]->evaluate_or(block, sel, selected_size, flags);
+ } else {
+ bool new_flags[selected_size];
+ memset(new_flags, true, selected_size);
+
+ for (auto block_column_predicate : _block_column_predicate_vec) {
+ block_column_predicate->evaluate_and(block, sel, selected_size, new_flags);
+ }
+
+ for (uint16_t i = 0; i < selected_size; i++) {
+ flags[i] |= new_flags[i];
+ }
+ }
+}
+
+// todo(wb) Can the 'and' of multiple bitmaps be vectorized?
+void AndBlockColumnPredicate::evaluate_vec(vectorized::MutableColumns& block, uint16_t size, bool* flags) const {
+ if (num_of_column_predicate() == 1) {
+ _block_column_predicate_vec[0]->evaluate_vec(block, size, flags);
+ } else {
+ bool new_flags[size];
+ for (auto block_column_predicate : _block_column_predicate_vec) {
+ memset(new_flags, true, size);
+ block_column_predicate->evaluate_vec(block, size, new_flags);
+
+ for (uint16_t j = 0; j < size; j++) {
+ flags[j] &= new_flags[j] ;
+ }
+ }
+ }
+}
+
} // namespace doris
diff --git a/be/src/olap/block_column_predicate.h b/be/src/olap/block_column_predicate.h
index 4c06486..ef2a4ca 100644
--- a/be/src/olap/block_column_predicate.h
+++ b/be/src/olap/block_column_predicate.h
@@ -44,6 +44,12 @@ public:
virtual void evaluate_or(RowBlockV2* block, uint16_t selected_size, bool* flags) const = 0;
virtual void get_all_column_ids(std::set<ColumnId>& column_id_set) const = 0;
+
+ virtual void evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const {};
+ virtual void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {};
+ virtual void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const {};
+
+ virtual void evaluate_vec(vectorized::MutableColumns& block, uint16_t size, bool* flags) const {};
};
class SingleColumnBlockPredicate : public BlockColumnPredicate {
@@ -57,6 +63,13 @@ public:
void get_all_column_ids(std::set<ColumnId>& column_id_set) const override {
column_id_set.insert(_predicate->column_id());
};
+
+ void evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const override;
+ void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const override;
+ void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const override;
+
+ void evaluate_vec(vectorized::MutableColumns& block, uint16_t size, bool* flags) const override;
+
private:
const ColumnPredicate* _predicate;
};
@@ -98,6 +111,12 @@ public:
// 2.Do AND SEMANTICS in flags use 1 result to get proper select flags
void evaluate_and(RowBlockV2* block, uint16_t selected_size, bool* flags) const override;
void evaluate_or(RowBlockV2* block, uint16_t selected_size, bool* flags) const override;
+
+ void evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const override;
+ void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const override;
+ void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const override;
+
+ // note(wb) we didnt't impelment evaluate_vec method here, because storage layer only support AND predicate now;
};
class AndBlockColumnPredicate : public MutilColumnBlockPredicate {
@@ -109,6 +128,13 @@ public:
// 1.AndBlockColumnPredicate need evaluate all child BlockColumnPredicate AND SEMANTICS inside first
// 2.Evaluate OR SEMANTICS in flags use 1 result to get proper select flags
void evaluate_or(RowBlockV2* block, uint16_t selected_size, bool* flags) const override;
+
+ void evaluate(vectorized::MutableColumns& block, uint16_t* sel, uint16_t* selected_size) const override;
+ void evaluate_and(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const override;
+ void evaluate_or(vectorized::MutableColumns& block, uint16_t* sel, uint16_t selected_size, bool* flags) const override;
+
+ void evaluate_vec(vectorized::MutableColumns& block, uint16_t size, bool* flags) const override;
+
};
} //namespace doris
diff --git a/be/src/olap/collect_iterator.cpp b/be/src/olap/collect_iterator.cpp
index 903ed0a..46a7aa4 100644
--- a/be/src/olap/collect_iterator.cpp
+++ b/be/src/olap/collect_iterator.cpp
@@ -32,7 +32,7 @@ CollectIterator::~CollectIterator() = default;
void CollectIterator::init(TabletReader* reader) {
_reader = reader;
// when aggregate is enabled or key_type is DUP_KEYS, we don't merge
- // multiple data to aggregate for performance in user fetch
+ // multiple data to aggregate for better performance
if (_reader->_reader_type == READER_QUERY &&
(_reader->_aggregation || _reader->_tablet->keys_type() == KeysType::DUP_KEYS)) {
_merge = false;
diff --git a/be/src/olap/column_predicate.h b/be/src/olap/column_predicate.h
index 21b115e..10b8a91 100644
--- a/be/src/olap/column_predicate.h
+++ b/be/src/olap/column_predicate.h
@@ -23,6 +23,7 @@
#include "olap/column_block.h"
#include "olap/rowset/segment_v2/bitmap_index_reader.h"
#include "olap/selection_vector.h"
+#include "vec/columns/column.h"
using namespace doris::segment_v2;
@@ -54,6 +55,18 @@ public:
const std::vector<BitmapIndexIterator*>& iterators, uint32_t num_rows,
roaring::Roaring* roaring) const = 0;
+ // evaluate predicate on IColumn
+ // a short circuit eval way
+ virtual void evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const {};
+ virtual void evaluate_and(vectorized::IColumn& column, uint16_t* sel, uint16_t size,
+ bool* flags) const {};
+ virtual void evaluate_or(vectorized::IColumn& column, uint16_t* sel, uint16_t size,
+ bool* flags) const {};
+
+ // used to evaluate pre read column in lazy matertialization
+ // now only support integer/float
+ // a vectorized eval way
+ virtual void evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const {};
uint32_t column_id() const { return _column_id; }
protected:
diff --git a/be/src/olap/comparison_predicate.cpp b/be/src/olap/comparison_predicate.cpp
index 7dbe7cb..598e7f3 100644
--- a/be/src/olap/comparison_predicate.cpp
+++ b/be/src/olap/comparison_predicate.cpp
@@ -21,6 +21,9 @@
#include "olap/schema.h"
#include "runtime/string_value.hpp"
#include "runtime/vectorized_row_batch.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/predicate_column.h"
namespace doris {
@@ -142,6 +145,75 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate, <=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define COMPARISON_PRED_COLUMN_EVALUATE(CLASS, OP) \
+ template <class type> \
+ void CLASS<type>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const { \
+ uint16_t new_size = 0; \
+ if (column.is_nullable()) { \
+ auto* nullable_column = vectorized::check_and_get_column<vectorized::ColumnNullable>(column);\
+ auto& null_bitmap = reinterpret_cast<const vectorized::ColumnVector<uint8_t>&>(*(nullable_column->get_null_map_column_ptr())).get_data(); \
+ auto* nest_column_vector = vectorized::check_and_get_column<vectorized::PredicateColumnType<type>>(nullable_column->get_nested_column());\
+ auto& data_array = nest_column_vector->get_data(); \
+ for (uint16_t i = 0; i < *size; i++) { \
+ uint16_t idx = sel[i]; \
+ sel[new_size] = idx; \
+ const type& cell_value = reinterpret_cast<const type&>(data_array[idx]); \
+ bool ret = !null_bitmap[idx] && (cell_value OP _value); \
+ new_size += _opposite ? !ret : ret; \
+ } \
+ *size = new_size; \
+ } else {\
+ auto& pred_column_ref = reinterpret_cast<vectorized::PredicateColumnType<type>&>(column);\
+ auto& data_array = pred_column_ref.get_data(); \
+ for (uint16_t i = 0; i < *size; i++) { \
+ uint16_t idx = sel[i]; \
+ sel[new_size] = idx; \
+ const type& cell_value = reinterpret_cast<const type&>(data_array[idx]); \
+ auto ret = cell_value OP _value; \
+ new_size += _opposite ? !ret : ret; \
+ } \
+ *size = new_size; \
+ }\
+ }
+
+COMPARISON_PRED_COLUMN_EVALUATE(EqualPredicate, ==)
+COMPARISON_PRED_COLUMN_EVALUATE(NotEqualPredicate, !=)
+COMPARISON_PRED_COLUMN_EVALUATE(LessPredicate, <)
+COMPARISON_PRED_COLUMN_EVALUATE(LessEqualPredicate, <=)
+COMPARISON_PRED_COLUMN_EVALUATE(GreaterPredicate, >)
+COMPARISON_PRED_COLUMN_EVALUATE(GreaterEqualPredicate, >=)
+
+#define COMPARISON_PRED_COLUMN_EVALUATE_VEC(CLASS, OP) \
+ template <class type> \
+ void CLASS<type>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const { \
+ if (column.is_nullable()) { \
+ auto* nullable_column = vectorized::check_and_get_column<vectorized::ColumnNullable>(column); \
+ auto& data_array = reinterpret_cast<const vectorized::ColumnVector<type>&>(nullable_column->get_nested_column()).get_data(); \
+ auto& null_bitmap = reinterpret_cast<const vectorized::ColumnVector<uint8_t>&>(*(nullable_column->get_null_map_column_ptr())).get_data(); \
+ for (uint16_t i = 0; i < size; i++) { \
+ flags[i] = (data_array[i] OP _value) && (!null_bitmap[i]); \
+ } \
+ } else { \
+ auto& predicate_column = reinterpret_cast<vectorized::PredicateColumnType<type>&>(column); \
+ auto& data_array = predicate_column.get_data(); \
+ for (uint16_t i = 0; i < size; i++) { \
+ flags[i] = data_array[i] OP _value; \
+ } \
+ } \
+ if (_opposite) { \
+ for (uint16_t i = 0; i < size; i++) { \
+ flags[i] = !flags[i]; \
+ } \
+ } \
+ }
+
+COMPARISON_PRED_COLUMN_EVALUATE_VEC(EqualPredicate, ==)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC(NotEqualPredicate, !=)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC(LessPredicate, <)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC(LessEqualPredicate, <=)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC(GreaterPredicate, >)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC(GreaterEqualPredicate, >=)
+
#define COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_OR(CLASS, OP) \
template <class type> \
void CLASS<type>::evaluate_or(ColumnBlock* block, uint16_t* sel, uint16_t size, bool* flags) \
@@ -174,6 +246,20 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_OR(LessEqualPredicate, <=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_OR(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_OR(GreaterEqualPredicate, >=)
+// todo(wb) support it
+#define COMPARISON_PRED_COLUMN_EVALUATE_OR(CLASS, OP) \
+ template <class type> \
+ void CLASS<type>::evaluate_or(vectorized::IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const { \
+ \
+ }
+
+COMPARISON_PRED_COLUMN_EVALUATE_OR(EqualPredicate, ==)
+COMPARISON_PRED_COLUMN_EVALUATE_OR(NotEqualPredicate, !=)
+COMPARISON_PRED_COLUMN_EVALUATE_OR(LessPredicate, <)
+COMPARISON_PRED_COLUMN_EVALUATE_OR(LessEqualPredicate, <=)
+COMPARISON_PRED_COLUMN_EVALUATE_OR(GreaterPredicate, >)
+COMPARISON_PRED_COLUMN_EVALUATE_OR(GreaterEqualPredicate, >=)
+
#define COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_AND(CLASS, OP) \
template <class type> \
void CLASS<type>::evaluate_and(ColumnBlock* block, uint16_t* sel, uint16_t size, bool* flags) \
@@ -206,6 +292,21 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_AND(LessEqualPredicate, <=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_AND(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE_AND(GreaterEqualPredicate, >=)
+//todo(wb) support it
+#define COMPARISON_PRED_COLUMN_EVALUATE_AND(CLASS, OP) \
+ template <class type> \
+ void CLASS<type>::evaluate_and(vectorized::IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const { \
+ \
+ \
+ }
+
+COMPARISON_PRED_COLUMN_EVALUATE_AND(EqualPredicate, ==)
+COMPARISON_PRED_COLUMN_EVALUATE_AND(NotEqualPredicate, !=)
+COMPARISON_PRED_COLUMN_EVALUATE_AND(LessPredicate, <)
+COMPARISON_PRED_COLUMN_EVALUATE_AND(LessEqualPredicate, <=)
+COMPARISON_PRED_COLUMN_EVALUATE_AND(GreaterPredicate, >)
+COMPARISON_PRED_COLUMN_EVALUATE_AND(GreaterEqualPredicate, >=)
+
#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal, iterator, bitmap, roaring) \
do { \
if (!s.is_not_found()) { \
@@ -440,4 +541,66 @@ COMPARISON_PRED_BITMAP_EVALUATE_DECLARATION(LessEqualPredicate)
COMPARISON_PRED_BITMAP_EVALUATE_DECLARATION(GreaterPredicate)
COMPARISON_PRED_BITMAP_EVALUATE_DECLARATION(GreaterEqualPredicate)
+#define COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(CLASS) \
+ template void CLASS<int8_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<int16_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<int32_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<int64_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<int128_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<float>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const; \
+ template void CLASS<double>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<decimal12_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<StringValue>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<uint24_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<uint64_t>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) \
+ const; \
+ template void CLASS<bool>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const;
+
+COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(EqualPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(NotEqualPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(LessPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(LessEqualPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(GreaterPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_DECLARATION(GreaterEqualPredicate)
+
+#define COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(CLASS) \
+ template void CLASS<int8_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<int16_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<int32_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<int64_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<int128_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<float>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const; \
+ template void CLASS<double>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<decimal12_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<StringValue>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<uint24_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<uint64_t>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) \
+ const; \
+ template void CLASS<bool>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const;
+
+COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(EqualPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(NotEqualPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(LessPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(LessEqualPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(GreaterPredicate)
+COMPARISON_PRED_COLUMN_EVALUATE_VEC_DECLARATION(GreaterEqualPredicate)
+
} //namespace doris
diff --git a/be/src/olap/comparison_predicate.h b/be/src/olap/comparison_predicate.h
index d0a4049..30fd9fd 100644
--- a/be/src/olap/comparison_predicate.h
+++ b/be/src/olap/comparison_predicate.h
@@ -40,6 +40,10 @@ class VectorizedRowBatch;
virtual Status evaluate(const Schema& schema, \
const std::vector<BitmapIndexIterator*>& iterators, \
uint32_t num_rows, roaring::Roaring* roaring) const override; \
+ void evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const override; \
+ void evaluate_and(vectorized::IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const override; \
+ void evaluate_or(vectorized::IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const override; \
+ void evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const override; \
\
private: \
type _value; \
diff --git a/be/src/olap/generic_iterators.cpp b/be/src/olap/generic_iterators.cpp
index adac5df..8a11108 100644
--- a/be/src/olap/generic_iterators.cpp
+++ b/be/src/olap/generic_iterators.cpp
@@ -116,6 +116,11 @@ public:
MergeIteratorContext(RowwiseIterator* iter, std::shared_ptr<MemTracker> parent)
: _iter(iter), _block(iter->schema(), 1024, std::move(parent)) {}
+ MergeIteratorContext(const MergeIteratorContext&) = delete;
+ MergeIteratorContext(MergeIteratorContext&&) = delete;
+ MergeIteratorContext& operator=(const MergeIteratorContext&) = delete;
+ MergeIteratorContext& operator=(MergeIteratorContext&&) = delete;
+
~MergeIteratorContext() {
delete _iter;
_iter = nullptr;
@@ -151,6 +156,7 @@ private:
private:
RowwiseIterator* _iter;
+
// used to store data load from iterator
RowBlockV2 _block;
@@ -181,10 +187,9 @@ Status MergeIteratorContext::advance() {
}
Status MergeIteratorContext::_load_next_block() {
- Status st;
do {
_block.clear();
- st = _iter->next_batch(&_block);
+ Status st = _iter->next_batch(&_block);
if (!st.ok()) {
_valid = false;
if (st.is_end_of_file()) {
@@ -202,27 +207,29 @@ Status MergeIteratorContext::_load_next_block() {
class MergeIterator : public RowwiseIterator {
public:
// MergeIterator takes the ownership of input iterators
- MergeIterator(std::list<RowwiseIterator*> iters, std::shared_ptr<MemTracker> parent, int sequence_id_idx)
- : _origin_iters(std::move(iters)), _sequence_id_idx(sequence_id_idx) {
+ MergeIterator(std::vector<RowwiseIterator*> iters, std::shared_ptr<MemTracker> parent, int sequence_id_idx)
+ : _origin_iters(std::move(iters)), _sequence_id_idx(sequence_id_idx), _merge_heap(MergeContextComparator(_sequence_id_idx)) {
// use for count the mem use of Block use in Merge
_mem_tracker = MemTracker::CreateTracker(-1, "MergeIterator", parent, false);
}
~MergeIterator() override {
- while (!_merge_heap->empty()) {
- auto ctx = _merge_heap->top();
- _merge_heap->pop();
+ while (!_merge_heap.empty()) {
+ auto ctx = _merge_heap.top();
+ _merge_heap.pop();
delete ctx;
}
}
+
Status init(const StorageReadOptions& opts) override;
+
Status next_batch(RowBlockV2* block) override;
const Schema& schema() const override { return *_schema; }
private:
// It will be released after '_merge_heap' has been built.
- std::list<RowwiseIterator*> _origin_iters;
+ std::vector<RowwiseIterator*> _origin_iters;
int _sequence_id_idx;
@@ -256,9 +263,12 @@ private:
int sequence_id_idx;
};
- using MergeHeap = std::priority_queue<MergeIteratorContext*, std::vector<MergeIteratorContext*>,
- MergeContextComparator>;
- std::unique_ptr<MergeHeap> _merge_heap;
+
+ using MergeHeap = std::priority_queue<MergeIteratorContext*,
+ std::vector<MergeIteratorContext*>,
+ MergeContextComparator>;
+
+ MergeHeap _merge_heap;
};
Status MergeIterator::init(const StorageReadOptions& opts) {
@@ -267,24 +277,24 @@ Status MergeIterator::init(const StorageReadOptions& opts) {
}
_schema.reset(new Schema((*(_origin_iters.begin()))->schema()));
- _merge_heap = std::make_unique<MergeHeap>(MergeContextComparator(_sequence_id_idx));
for (auto iter : _origin_iters) {
std::unique_ptr<MergeIteratorContext> ctx(new MergeIteratorContext(iter, _mem_tracker));
RETURN_IF_ERROR(ctx->init(opts));
if (!ctx->valid()) {
continue;
}
- _merge_heap->push(ctx.release());
+ _merge_heap.push(ctx.release());
}
+
_origin_iters.clear();
return Status::OK();
}
Status MergeIterator::next_batch(RowBlockV2* block) {
size_t row_idx = 0;
- for (; row_idx < block->capacity() && !_merge_heap->empty(); ++row_idx) {
- auto ctx = _merge_heap->top();
- _merge_heap->pop();
+ for (; row_idx < block->capacity() && !_merge_heap.empty(); ++row_idx) {
+ auto ctx = _merge_heap.top();
+ _merge_heap.pop();
RowBlockRow dst_row = block->row(row_idx);
// copy current row to block
@@ -292,7 +302,7 @@ Status MergeIterator::next_batch(RowBlockV2* block) {
RETURN_IF_ERROR(ctx->advance());
if (ctx->valid()) {
- _merge_heap->push(ctx);
+ _merge_heap.push(ctx);
} else {
// Release ctx earlier to reduce resource consumed
delete ctx;
@@ -313,17 +323,17 @@ public:
// Iterators' ownership it transfered to this class.
// This class will delete all iterators when destructs
// Client should not use iterators any more.
- UnionIterator(std::list<RowwiseIterator*> iters, std::shared_ptr<MemTracker> parent)
- : _origin_iters(std::move(iters)) {
+ UnionIterator(std::vector<RowwiseIterator*> &v, std::shared_ptr<MemTracker> parent)
+ : _origin_iters(v.begin(), v.end()) {
_mem_tracker = MemTracker::CreateTracker(-1, "UnionIterator", parent, false);
}
~UnionIterator() override {
- for (auto iter : _origin_iters) {
- delete iter;
- }
+ std::for_each(_origin_iters.begin(), _origin_iters.end(), std::default_delete<RowwiseIterator>());
}
+
Status init(const StorageReadOptions& opts) override;
+
Status next_batch(RowBlockV2* block) override;
const Schema& schema() const override { return *_schema; }
@@ -331,7 +341,7 @@ public:
private:
std::unique_ptr<Schema> _schema;
RowwiseIterator* _cur_iter = nullptr;
- std::list<RowwiseIterator*> _origin_iters;
+ std::deque<RowwiseIterator*> _origin_iters;
};
Status UnionIterator::init(const StorageReadOptions& opts) {
@@ -364,18 +374,18 @@ Status UnionIterator::next_batch(RowBlockV2* block) {
return Status::EndOfFile("End of UnionIterator");
}
-RowwiseIterator* new_merge_iterator(std::list<RowwiseIterator*> inputs, std::shared_ptr<MemTracker> parent, int sequence_id_idx) {
+RowwiseIterator* new_merge_iterator(std::vector<RowwiseIterator*> inputs, std::shared_ptr<MemTracker> parent, int sequence_id_idx) {
if (inputs.size() == 1) {
return *(inputs.begin());
}
return new MergeIterator(std::move(inputs), parent, sequence_id_idx);
}
-RowwiseIterator* new_union_iterator(std::list<RowwiseIterator*> inputs, std::shared_ptr<MemTracker> parent) {
+RowwiseIterator* new_union_iterator(std::vector<RowwiseIterator*>& inputs, std::shared_ptr<MemTracker> parent) {
if (inputs.size() == 1) {
return *(inputs.begin());
}
- return new UnionIterator(std::move(inputs), parent);
+ return new UnionIterator(inputs, parent);
}
RowwiseIterator* new_auto_increment_iterator(const Schema& schema, size_t num_rows) {
diff --git a/be/src/olap/generic_iterators.h b/be/src/olap/generic_iterators.h
index 1d3eccf..e8f4528 100644
--- a/be/src/olap/generic_iterators.h
+++ b/be/src/olap/generic_iterators.h
@@ -25,14 +25,14 @@ namespace doris {
//
// Inputs iterators' ownership is taken by created merge iterator. And client
// should delete returned iterator after usage.
-RowwiseIterator* new_merge_iterator(std::list<RowwiseIterator*> inputs, std::shared_ptr<MemTracker> parent, int sequence_id_idx);
+RowwiseIterator* new_merge_iterator(std::vector<RowwiseIterator*> inputs, std::shared_ptr<MemTracker> parent, int sequence_id_idx);
// Create a union iterator for input iterators. Union iterator will read
// input iterators one by one.
//
// Inputs iterators' ownership is taken by created union iterator. And client
// should delete returned iterator after usage.
-RowwiseIterator* new_union_iterator(std::list<RowwiseIterator*> inputs, std::shared_ptr<MemTracker> parent);
+RowwiseIterator* new_union_iterator(std::vector<RowwiseIterator*>& inputs, std::shared_ptr<MemTracker> parent);
// Create an auto increment iterator which returns num_rows data in format of schema.
// This class aims to be used in unit test.
diff --git a/be/src/olap/iterators.h b/be/src/olap/iterators.h
index d514dc0..4cdfc60 100644
--- a/be/src/olap/iterators.h
+++ b/be/src/olap/iterators.h
@@ -23,6 +23,7 @@
#include "olap/olap_common.h"
#include "olap/column_predicate.h"
#include "olap/block_column_predicate.h"
+#include "vec/core/block.h"
namespace doris {
@@ -80,6 +81,7 @@ public:
// REQUIRED (null is not allowed)
OlapReaderStatistics* stats = nullptr;
bool use_page_cache = false;
+ int block_row_max = 4096;
};
// Used to read data in RowBlockV2 one by one
@@ -99,7 +101,9 @@ public:
// into input batch with Status::OK() returned
// If there is no data to read, will return Status::EndOfFile.
// If other error happens, other error code will be returned.
- virtual Status next_batch(RowBlockV2* block) = 0;
+ virtual Status next_batch(RowBlockV2* block) { return Status::NotSupported("to be implemented"); }
+
+ virtual Status next_batch(vectorized::Block* block) { return Status::NotSupported("to be implemented"); }
// return schema for this Iterator
virtual const Schema& schema() const = 0;
diff --git a/be/src/olap/null_predicate.cpp b/be/src/olap/null_predicate.cpp
index cc88d26..da3eb29 100644
--- a/be/src/olap/null_predicate.cpp
+++ b/be/src/olap/null_predicate.cpp
@@ -20,6 +20,9 @@
#include "olap/field.h"
#include "runtime/string_value.hpp"
#include "runtime/vectorized_row_batch.h"
+#include "vec/columns/column_nullable.h"
+
+using namespace doris::vectorized;
namespace doris {
@@ -118,4 +121,44 @@ Status NullPredicate::evaluate(const Schema& schema,
return Status::OK();
}
+void NullPredicate::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const {
+ uint16_t new_size = 0;
+ if (auto* nullable = check_and_get_column<ColumnNullable>(column)) {
+ auto& null_map = nullable->get_null_map_data();
+ for (uint16_t i = 0; i < *size; ++i) {
+ uint16_t idx = sel[i];
+ sel[new_size] = idx;
+ new_size += (null_map[idx] == _is_null);
+ }
+ *size = new_size;
+ } else {
+ if (_is_null) *size = 0;
+ }
+}
+
+void NullPredicate::evaluate_or(IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const {
+ if (auto* nullable = check_and_get_column<ColumnNullable>(column)) {
+ auto& null_map = nullable->get_null_map_data();
+ for (uint16_t i = 0; i < size; ++i) {
+ if (flags[i]) continue;
+ uint16_t idx = sel[i];
+ flags[i] |= (null_map[idx] == _is_null);
+ }
+ } else {
+ if (!_is_null) memset(flags, true, size);
+ }
+}
+
+void NullPredicate::evaluate_and(IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const {
+ if (auto* nullable = check_and_get_column<ColumnNullable>(column)) {
+ auto& null_map = nullable->get_null_map_data();
+ for (uint16_t i = 0; i < size; ++i) {
+ if (flags[i]) continue;
+ uint16_t idx = sel[i];
+ flags[i] &= (null_map[idx] == _is_null);
+ }
+ } else {
+ if (_is_null) memset(flags, false, size);
+ }
+}
} //namespace doris
diff --git a/be/src/olap/null_predicate.h b/be/src/olap/null_predicate.h
index e867d58..681e60b 100644
--- a/be/src/olap/null_predicate.h
+++ b/be/src/olap/null_predicate.h
@@ -43,6 +43,14 @@ public:
virtual Status evaluate(const Schema& schema, const vector<BitmapIndexIterator*>& iterators,
uint32_t num_rows, roaring::Roaring* roaring) const override;
+ void evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const override;
+
+ void evaluate_or(vectorized::IColumn& column, uint16_t* sel, uint16_t size,
+ bool* flags) const override;
+
+ void evaluate_and(vectorized::IColumn& column, uint16_t* sel, uint16_t size,
+ bool* flags) const override;
+
private:
bool _is_null; //true for null, false for not null
};
diff --git a/be/src/olap/reader.cpp b/be/src/olap/reader.cpp
index 763d1e7..13e50b8 100644
--- a/be/src/olap/reader.cpp
+++ b/be/src/olap/reader.cpp
@@ -41,6 +41,7 @@
#include "runtime/string_value.hpp"
#include "util/date_func.h"
#include "util/mem_util.hpp"
+#include "vec/olap/vcollect_iterator.h"
using std::nothrow;
using std::set;
@@ -57,8 +58,8 @@ void TabletReader::ReaderParams::check_validation() const {
std::string TabletReader::ReaderParams::to_string() const {
std::stringstream ss;
ss << "tablet=" << tablet->full_name() << " reader_type=" << reader_type
- << " aggregation=" << aggregation << " version=" << version << " start_key_include=" << start_key_include
- << " end_key_include=" << end_key_include;
+ << " aggregation=" << aggregation << " version=" << version
+ << " start_key_include=" << start_key_include << " end_key_include=" << end_key_include;
for (const auto& key : start_key) {
ss << " keys=" << key;
@@ -230,6 +231,7 @@ OLAPStatus TabletReader::_capture_rs_readers(const ReaderParams& read_params,
OLAPStatus TabletReader::_init_params(const ReaderParams& read_params) {
read_params.check_validation();
+ _direct_mode = read_params.direct_mode;
_aggregation = read_params.aggregation;
_need_agg_finalize = read_params.need_agg_finalize;
_reader_type = read_params.reader_type;
@@ -699,7 +701,7 @@ ColumnPredicate* TabletReader::_parse_to_predicate(const TCondition& condition,
break;
}
case OLAP_FIELD_TYPE_VARCHAR:
- case OLAP_FIELD_TYPE_STRING:{
+ case OLAP_FIELD_TYPE_STRING: {
phmap::flat_hash_set<StringValue> values;
for (auto& cond_val : condition.condition_values) {
StringValue value;
@@ -806,7 +808,8 @@ void TabletReader::_init_load_bf_columns(const ReaderParams& read_params, Condit
return;
}
FieldType type = _tablet->tablet_schema().column(max_equal_index).type();
- if ((type != OLAP_FIELD_TYPE_VARCHAR && type != OLAP_FIELD_TYPE_STRING)|| max_equal_index + 1 > _tablet->num_short_key_columns()) {
+ if ((type != OLAP_FIELD_TYPE_VARCHAR && type != OLAP_FIELD_TYPE_STRING) ||
+ max_equal_index + 1 > _tablet->num_short_key_columns()) {
load_bf_columns->erase(max_equal_index);
}
}
diff --git a/be/src/olap/reader.h b/be/src/olap/reader.h
index 0f20895..82cd7ff 100644
--- a/be/src/olap/reader.h
+++ b/be/src/olap/reader.h
@@ -49,6 +49,11 @@ class RowBlock;
class CollectIterator;
class RuntimeState;
+namespace vectorized {
+class VCollectIterator;
+class Block;
+} // namespace vectorized
+
class TabletReader {
struct KeysParam {
std::string to_string() const;
@@ -64,6 +69,7 @@ public:
struct ReaderParams {
TabletSharedPtr tablet;
ReaderType reader_type = READER_QUERY;
+ bool direct_mode = false;
bool aggregation = false;
bool need_agg_finalize = true;
// 1. when read column data page:
@@ -88,6 +94,9 @@ public:
RuntimeProfile* profile = nullptr;
RuntimeState* runtime_state = nullptr;
+ // use only in vec exec engine
+ std::vector<uint32_t>* origin_return_columns = nullptr;
+
void check_validation() const;
std::string to_string() const;
@@ -105,7 +114,17 @@ public:
// Return OLAP_SUCCESS and set `*eof` to true when no more rows can be read.
// Return others when unexpected error happens.
virtual OLAPStatus next_row_with_aggregation(RowCursor* row_cursor, MemPool* mem_pool,
- ObjectPool* agg_pool, bool* eof) = 0;
+ ObjectPool* agg_pool, bool* eof) = 0;
+
+ // Read next block with aggregation.
+ // Return OLAP_SUCCESS and set `*eof` to false when next block is read
+ // Return OLAP_SUCCESS and set `*eof` to true when no more rows can be read.
+ // Return others when unexpected error happens.
+ // TODO: Rethink here we still need mem_pool and agg_pool?
+ virtual OLAPStatus next_block_with_aggregation(vectorized::Block* block, MemPool* mem_pool,
+ ObjectPool* agg_pool, bool* eof) {
+ return OLAP_ERR_READER_INITIALIZE_ERROR;
+ }
uint64_t merged_rows() const { return _merged_rows; }
@@ -119,6 +138,7 @@ public:
protected:
friend class CollectIterator;
+ friend class vectorized::VCollectIterator;
friend class DeleteHandler;
OLAPStatus _init_params(const ReaderParams& read_params);
@@ -188,8 +208,8 @@ protected:
ReaderType _reader_type = READER_QUERY;
bool _next_delete_flag = false;
bool _filter_delete = false;
- bool _has_sequence_col = false;
int32_t _sequence_col_idx = -1;
+ bool _direct_mode = false;
CollectIterator _collect_iter;
std::vector<uint32_t> _key_cids;
diff --git a/be/src/olap/row_block2.cpp b/be/src/olap/row_block2.cpp
index 20ed4ae..26b58ca 100644
--- a/be/src/olap/row_block2.cpp
+++ b/be/src/olap/row_block2.cpp
@@ -23,6 +23,11 @@
#include "gutil/strings/substitute.h"
#include "olap/row_cursor.h"
#include "util/bitmap.h"
+#include "vec/columns/column_complex.h"
+#include "vec/columns/column_vector.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#include "vec/runtime/vdatetime_value.h"
using strings::Substitute;
namespace doris {
@@ -90,6 +95,209 @@ Status RowBlockV2::convert_to_row_block(RowCursor* helper, RowBlock* dst) {
return Status::OK();
}
+void RowBlockV2::_copy_data_to_column(int cid, doris::vectorized::MutableColumnPtr& origin_column) {
+ auto* column = origin_column.get();
+ bool nullable_mark_array[_selected_size];
+
+ bool column_nullable = origin_column->is_nullable();
+ bool origin_nullable = _schema.column(cid)->is_nullable();
+ if (column_nullable) {
+ auto nullable_column = assert_cast<vectorized::ColumnNullable*>(origin_column.get());
+ auto& null_map = nullable_column->get_null_map_data();
+ column = nullable_column->get_nested_column_ptr().get();
+
+ if (origin_nullable) {
+ for (uint16_t i = 0; i < _selected_size; ++i) {
+ uint16_t row_idx = _selection_vector[i];
+ null_map.push_back(_column_vector_batches[cid]->is_null_at(row_idx));
+ nullable_mark_array[i] = null_map.back();
+ }
+ } else {
+ null_map.resize_fill(null_map.size() + _selected_size, 0);
+ memset(nullable_mark_array, false, _selected_size * sizeof(bool));
+ }
+ } else {
+ memset(nullable_mark_array, false, _selected_size * sizeof(bool));
+ }
+
+ auto insert_data_directly = [this, &nullable_mark_array](int cid, auto& column) {
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ column->insert_data(
+ reinterpret_cast<const char*>(column_block(cid).cell_ptr(row_idx)), 0);
+ } else {
+ column->insert_default();
+ }
+ }
+ };
+
+ switch (_schema.column(cid)->type()) {
+ case OLAP_FIELD_TYPE_OBJECT: {
+ auto column_bitmap = assert_cast<vectorized::ColumnBitmap*>(column);
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ column_bitmap->insert_default();
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto slice = reinterpret_cast<const Slice*>(column_block(cid).cell_ptr(row_idx));
+
+ BitmapValue* pvalue = &column_bitmap->get_element(column_bitmap->size() - 1);
+
+ if (slice->size != 0) {
+ BitmapValue value;
+ value.deserialize(slice->data);
+ *pvalue = std::move(value);
+ } else {
+ *pvalue = std::move(*reinterpret_cast<BitmapValue*>(slice->data));
+ }
+ }
+ }
+ break;
+ }
+ case OLAP_FIELD_TYPE_HLL:
+ case OLAP_FIELD_TYPE_MAP:
+ case OLAP_FIELD_TYPE_VARCHAR: {
+ auto column_string = assert_cast<vectorized::ColumnString*>(column);
+
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto slice = reinterpret_cast<const Slice*>(column_block(cid).cell_ptr(row_idx));
+ column_string->insert_data(slice->data, slice->size);
+ } else {
+ column_string->insert_default();
+ }
+ }
+ break;
+ }
+ case OLAP_FIELD_TYPE_CHAR: {
+ auto column_string = assert_cast<vectorized::ColumnString*>(column);
+
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto slice = reinterpret_cast<const Slice*>(column_block(cid).cell_ptr(row_idx));
+ column_string->insert_data(slice->data, strnlen(slice->data, slice->size));
+ } else {
+ column_string->insert_default();
+ }
+ }
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATE: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int64>*>(column);
+
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto ptr = reinterpret_cast<const char*>(column_block(cid).cell_ptr(row_idx));
+
+ uint64_t value = 0;
+ value = *(unsigned char*)(ptr + 2);
+ value <<= 8;
+ value |= *(unsigned char*)(ptr + 1);
+ value <<= 8;
+ value |= *(unsigned char*)(ptr);
+ vectorized::VecDateTimeValue date;
+ date.from_olap_date(value);
+ (column_int)->insert_data(reinterpret_cast<char*>(&date), 0);
+ } else
+ column_int->insert_default();
+ }
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIME: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int64>*>(column);
+
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto ptr = reinterpret_cast<const char*>(column_block(cid).cell_ptr(row_idx));
+
+ uint64_t value = *reinterpret_cast<const uint64_t*>(ptr);
+ vectorized::VecDateTimeValue data(value);
+ (column_int)->insert_data(reinterpret_cast<char*>(&data), 0);
+ } else {
+ column_int->insert_default();
+ }
+ }
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL: {
+ auto column_decimal =
+ assert_cast<vectorized::ColumnDecimal<vectorized::Decimal128>*>(column);
+
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto ptr = reinterpret_cast<const char*>(column_block(cid).cell_ptr(row_idx));
+
+ int64_t int_value = *(int64_t*)(ptr);
+ int32_t frac_value = *(int32_t*)(ptr + sizeof(int64_t));
+ DecimalV2Value data(int_value, frac_value);
+ column_decimal->insert_data(reinterpret_cast<char*>(&data), 0);
+ } else {
+ column_decimal->insert_default();
+ }
+ }
+ break;
+ }
+ case OLAP_FIELD_TYPE_INT: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int32>*>(column);
+ insert_data_directly(cid, column_int);
+ break;
+ }
+ case OLAP_FIELD_TYPE_BOOL: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::UInt8>*>(column);
+ insert_data_directly(cid, column_int);
+ break;
+ }
+ case OLAP_FIELD_TYPE_TINYINT: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int8>*>(column);
+ insert_data_directly(cid, column_int);
+ break;
+ }
+ case OLAP_FIELD_TYPE_SMALLINT: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int16>*>(column);
+ insert_data_directly(cid, column_int);
+ break;
+ }
+ case OLAP_FIELD_TYPE_BIGINT: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int64>*>(column);
+ insert_data_directly(cid, column_int);
+ break;
+ }
+ case OLAP_FIELD_TYPE_LARGEINT: {
+ auto column_int = assert_cast<vectorized::ColumnVector<vectorized::Int128>*>(column);
+ insert_data_directly(cid, column_int);
+ break;
+ }
+ case OLAP_FIELD_TYPE_FLOAT: {
+ auto column_float = assert_cast<vectorized::ColumnVector<vectorized::Float32>*>(column);
+ insert_data_directly(cid, column_float);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DOUBLE: {
+ auto column_float = assert_cast<vectorized::ColumnVector<vectorized::Float64>*>(column);
+ insert_data_directly(cid, column_float);
+ break;
+ }
+ default: {
+ DCHECK(false) << "Invalid type in RowBlockV2:" << _schema.column(cid)->type();
+ }
+ }
+}
+
+Status RowBlockV2::convert_to_vec_block(vectorized::Block* block) {
+ for (int i = 0; i < _schema.column_ids().size(); ++i) {
+ auto cid = _schema.column_ids()[i];
+ auto column = (*std::move(block->get_by_position(i).column)).assume_mutable();
+ _copy_data_to_column(cid, column);
+ }
+ _pool->clear();
+ return Status::OK();
+}
+
std::string RowBlockRow::debug_string() const {
std::stringstream ss;
ss << "[";
diff --git a/be/src/olap/row_block2.h b/be/src/olap/row_block2.h
index ece8d72..cdbf428 100644
--- a/be/src/olap/row_block2.h
+++ b/be/src/olap/row_block2.h
@@ -73,6 +73,9 @@ public:
// convert RowBlockV2 to RowBlock
Status convert_to_row_block(RowCursor* helper, RowBlock* dst);
+ // convert RowBlockV2 to vectorized::Block
+ Status convert_to_vec_block(vectorized::Block* block);
+
// low-level API to access memory for each column block(including data array and nullmap).
// `cid` must be one of `schema()->column_ids()`.
ColumnBlock column_block(ColumnId cid) const {
@@ -106,6 +109,8 @@ public:
std::string debug_string();
private:
+ void _copy_data_to_column(int cid, vectorized::MutableColumnPtr& mutable_column_ptr);
+
Schema _schema;
size_t _capacity;
// _column_vector_batches[cid] == null if cid is not in `_schema`.
diff --git a/be/src/olap/rowset/alpha_rowset_reader.h b/be/src/olap/rowset/alpha_rowset_reader.h
index 959a90b..fd90f37 100644
--- a/be/src/olap/rowset/alpha_rowset_reader.h
+++ b/be/src/olap/rowset/alpha_rowset_reader.h
@@ -65,6 +65,10 @@ public:
// It's ok, because we only get ref here, the block's owner is this reader.
OLAPStatus next_block(RowBlock** block) override;
+ OLAPStatus next_block(vectorized::Block *block) override {
+ return OLAP_ERR_DATA_EOF;
+ }
+
bool delete_flag() override;
Version version() override;
diff --git a/be/src/olap/rowset/beta_rowset_reader.cpp b/be/src/olap/rowset/beta_rowset_reader.cpp
index ada5157..459f3ca 100644
--- a/be/src/olap/rowset/beta_rowset_reader.cpp
+++ b/be/src/olap/rowset/beta_rowset_reader.cpp
@@ -27,6 +27,8 @@
#include "olap/rowset/segment_v2/segment_iterator.h"
#include "olap/schema.h"
+#include "vec/core/block.h"
+
namespace doris {
BetaRowsetReader::BetaRowsetReader(BetaRowsetSharedPtr rowset,
@@ -107,7 +109,8 @@ OLAPStatus BetaRowsetReader::init(RowsetReaderContext* read_context) {
}
seg_iterators.push_back(std::move(iter));
}
- std::list<RowwiseIterator*> iterators;
+
+ std::vector<RowwiseIterator*> iterators;
for (auto& owned_it : seg_iterators) {
// transfer ownership of segment iterator to `_iterator`
iterators.push_back(owned_it.release());
@@ -172,4 +175,41 @@ OLAPStatus BetaRowsetReader::next_block(RowBlock** block) {
return OLAP_SUCCESS;
}
+OLAPStatus BetaRowsetReader::next_block(vectorized::Block* block) {
+ SCOPED_RAW_TIMER(&_stats->block_fetch_ns);
+ bool is_first = true;
+
+ do {
+ // read next input block
+ {
+ _input_block->clear();
+ {
+ auto s = _iterator->next_batch(_input_block.get());
+ if (!s.ok()) {
+ if (s.is_end_of_file()) {
+ if (is_first) {
+ return OLAP_ERR_DATA_EOF;
+ } else {
+ break;
+ }
+ } else {
+ LOG(WARNING) << "failed to read next block: " << s.to_string();
+ return OLAP_ERR_ROWSET_READ_FAILED;
+ }
+ } else if (_input_block->selected_size() == 0) {
+ continue;
+ }
+ }
+ }
+
+ {
+ SCOPED_RAW_TIMER(&_stats->block_convert_ns);
+ _input_block->convert_to_vec_block(block);
+ }
+ is_first = false;
+ } while (block->rows() < _context->runtime_state->batch_size()); // here we should keep block.rows() < batch_size
+
+ return OLAP_SUCCESS;
+}
+
} // namespace doris
diff --git a/be/src/olap/rowset/beta_rowset_reader.h b/be/src/olap/rowset/beta_rowset_reader.h
index 34e7d9c..6074308 100644
--- a/be/src/olap/rowset/beta_rowset_reader.h
+++ b/be/src/olap/rowset/beta_rowset_reader.h
@@ -40,6 +40,7 @@ public:
// If parent_tracker is not null, the block we get from next_block() will have the parent_tracker.
// It's ok, because we only get ref here, the block's owner is this reader.
OLAPStatus next_block(RowBlock** block) override;
+ OLAPStatus next_block(vectorized::Block* block) override;
bool delete_flag() override { return _rowset->delete_flag(); }
diff --git a/be/src/olap/rowset/rowset_reader.h b/be/src/olap/rowset/rowset_reader.h
index b9e46d1..53af387 100644
--- a/be/src/olap/rowset/rowset_reader.h
+++ b/be/src/olap/rowset/rowset_reader.h
@@ -23,9 +23,14 @@
#include "olap/rowset/rowset.h"
#include "olap/rowset/rowset_reader_context.h"
+#include "vec/core/block.h"
namespace doris {
+namespace vectorized {
+class Block;
+}
+
class RowBlock;
class RowsetReader;
using RowsetReaderSharedPtr = std::shared_ptr<RowsetReader>;
@@ -33,6 +38,7 @@ using RowsetReaderSharedPtr = std::shared_ptr<RowsetReader>;
class RowsetReader {
public:
enum RowsetReaderType { ALPHA, BETA };
+
virtual ~RowsetReader() {}
// reader init
@@ -45,6 +51,8 @@ public:
// Others when error happens.
virtual OLAPStatus next_block(RowBlock** block) = 0;
+ virtual OLAPStatus next_block(vectorized::Block* block) = 0;
+
virtual bool delete_flag() = 0;
virtual Version version() = 0;
diff --git a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
index 6fae7eb..7825dcc 100644
--- a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
+++ b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
@@ -19,9 +19,12 @@
#include "common/logging.h"
#include "gutil/strings/substitute.h" // for Substitute
-#include "olap/rowset/segment_v2/bitshuffle_page.h"
#include "runtime/mem_pool.h"
#include "util/slice.h" // for Slice
+#include "vec/columns/column_vector.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/predicate_column.h"
namespace doris {
namespace segment_v2 {
@@ -220,6 +223,8 @@ Status BinaryDictPageDecoder::init() {
return Status::OK();
}
+BinaryDictPageDecoder::~BinaryDictPageDecoder() {}
+
Status BinaryDictPageDecoder::seek_to_position_in_page(size_t pos) {
return _data_page_decoder->seek_to_position_in_page(pos);
}
@@ -230,8 +235,53 @@ bool BinaryDictPageDecoder::is_dict_encoding() const {
void BinaryDictPageDecoder::set_dict_decoder(PageDecoder* dict_decoder) {
_dict_decoder = (BinaryPlainPageDecoder*)dict_decoder;
+ _bit_shuffle_ptr = reinterpret_cast<BitShufflePageDecoder<OLAP_FIELD_TYPE_INT>*>(_data_page_decoder.get());
};
+Status BinaryDictPageDecoder::next_batch(size_t* n, vectorized::MutableColumnPtr &dst) {
+ if (_encoding_type == PLAIN_ENCODING) {
+ return _data_page_decoder->next_batch(n, dst);
+ }
+ // dictionary encoding
+ DCHECK(_parsed);
+ DCHECK(_dict_decoder != nullptr) << "dict decoder pointer is nullptr";
+
+ if (PREDICT_FALSE(*n == 0 || _bit_shuffle_ptr->_cur_index >= _bit_shuffle_ptr->_num_elements)) {
+ *n = 0;
+ return Status::OK();
+ }
+
+ size_t max_fetch = std::min(*n, static_cast<size_t>(_bit_shuffle_ptr->_num_elements - _bit_shuffle_ptr->_cur_index));
+ *n = max_fetch;
+
+ const int32_t* data_array = reinterpret_cast<const int32_t*>(_bit_shuffle_ptr->_chunk.data);
+ size_t start_index = _bit_shuffle_ptr->_cur_index;
+
+ // todo(wb) support nullable
+ if (dst->is_predicate_column()) {
+ // cast columnptr to columnstringvalue just for avoid virtual function call overhead
+ vectorized::ColumnStringValue& string_value_vector = reinterpret_cast<vectorized::ColumnStringValue&>(*dst);
+ for (int i = 0; i < max_fetch; i++, start_index++) {
+ int32_t codeword = data_array[start_index];
+ uint32_t start_offset = _start_offset_array[codeword];
+ uint32_t str_len = _len_array[codeword];
+ string_value_vector.insert_data(&_dict_decoder->_data[start_offset], str_len);
+ }
+ } else {
+ // todo(wb) research whether using batch memcpy to insert columnString can has better performance when data set is big
+ for (int i = 0; i < max_fetch; i++, start_index++) {
+ int32_t codeword = data_array[start_index];
+ const uint32_t start_offset = _start_offset_array[codeword];
+ const uint32_t str_len = _len_array[codeword];
+ dst->insert_data(&_dict_decoder->_data[start_offset], str_len);
+ }
+ }
+ _bit_shuffle_ptr->_cur_index += max_fetch;
+
+ return Status::OK();
+
+}
+
Status BinaryDictPageDecoder::next_batch(size_t* n, ColumnBlockView* dst) {
if (_encoding_type == PLAIN_ENCODING) {
return _data_page_decoder->next_batch(n, dst);
diff --git a/be/src/olap/rowset/segment_v2/binary_dict_page.h b/be/src/olap/rowset/segment_v2/binary_dict_page.h
index 9290466..25a70b9 100644
--- a/be/src/olap/rowset/segment_v2/binary_dict_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_dict_page.h
@@ -33,6 +33,7 @@
#include "olap/types.h"
#include "runtime/mem_pool.h"
#include "runtime/mem_tracker.h"
+#include "olap/rowset/segment_v2/bitshuffle_page.h"
namespace doris {
namespace segment_v2 {
@@ -106,6 +107,8 @@ public:
Status next_batch(size_t* n, ColumnBlockView* dst) override;
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override;
+
size_t count() const override { return _data_page_decoder->count(); }
size_t current_index() const override { return _data_page_decoder->current_index(); }
@@ -114,15 +117,21 @@ public:
void set_dict_decoder(PageDecoder* dict_decoder);
+ ~BinaryDictPageDecoder();
+
private:
Slice _data;
PageDecoderOptions _options;
std::unique_ptr<PageDecoder> _data_page_decoder;
const BinaryPlainPageDecoder* _dict_decoder = nullptr;
+ BitShufflePageDecoder<OLAP_FIELD_TYPE_INT>* _bit_shuffle_ptr = nullptr;
bool _parsed;
EncodingTypePB _encoding_type;
// use as data buf.
std::unique_ptr<ColumnVectorBatch> _batch;
+
+ uint32_t* _start_offset_array = nullptr;
+ uint32_t* _len_array = nullptr;
};
} // namespace segment_v2
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 0747df2..25ce9e0 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -227,6 +227,35 @@ public:
return Status::OK();
}
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override {
+ DCHECK(_parsed);
+ if (PREDICT_FALSE(*n == 0 || _cur_idx >= _num_elems)) {
+ *n = 0;
+ return Status::OK();
+ }
+ const size_t max_fetch = std::min(*n, static_cast<size_t>(_num_elems - _cur_idx));
+
+ if (dst->is_predicate_column()) {
+ // todo(wb) padding sv here for better comparison performance
+ for (size_t i = 0; i < max_fetch; i++, _cur_idx++) {
+ const uint32_t start_offset = offset(_cur_idx);
+ uint32_t len = offset(_cur_idx + 1) - start_offset;
+ StringValue sv(const_cast<char*>(&_data[start_offset]), len);
+ dst->insert_data(reinterpret_cast<char*>(&sv), 0);
+ }
+ } else {
+ for (size_t i = 0; i < max_fetch; i++, _cur_idx++) {
+ // todo(wb) need more test case and then improve here
+ const uint32_t start_offset = offset(_cur_idx);
+ uint32_t len = offset(_cur_idx + 1) - start_offset;
+ dst->insert_data(&_data[start_offset], len);
+ }
+ }
+
+ *n = max_fetch;
+ return Status::OK();
+ };
+
size_t count() const override {
DCHECK(_parsed);
return _num_elems;
@@ -263,6 +292,8 @@ private:
// Index of the currently seeked element in the page.
uint32_t _cur_idx;
+ friend class BinaryDictPageDecoder;
+ friend class FileColumnIterator;
};
} // namespace segment_v2
diff --git a/be/src/olap/rowset/segment_v2/binary_prefix_page.h b/be/src/olap/rowset/segment_v2/binary_prefix_page.h
index 7e51b82..7d7bcc9 100644
--- a/be/src/olap/rowset/segment_v2/binary_prefix_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_prefix_page.h
@@ -113,6 +113,10 @@ public:
Status next_batch(size_t* n, ColumnBlockView* dst) override;
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override {
+ return Status::NotSupported("binary prefix page not implement vec op now");
+ };
+
size_t count() const override {
DCHECK(_parsed);
return _num_values;
diff --git a/be/src/olap/rowset/segment_v2/bitshuffle_page.h b/be/src/olap/rowset/segment_v2/bitshuffle_page.h
index cd54b6b..5afbb7d 100644
--- a/be/src/olap/rowset/segment_v2/bitshuffle_page.h
+++ b/be/src/olap/rowset/segment_v2/bitshuffle_page.h
@@ -348,6 +348,53 @@ public:
return Status::OK();
}
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override {
+ DCHECK(_parsed);
+ if (PREDICT_FALSE(*n == 0 || _cur_index >= _num_elements)) {
+ *n = 0;
+ return Status::OK();
+ }
+
+ size_t max_fetch = std::min(*n, static_cast<size_t>(_num_elements - _cur_index));
+
+ int begin = _cur_index;
+ int end = _cur_index + max_fetch;
+
+ // todo(wb) Try to eliminate type judgment in pagedecoder
+ if (dst->is_column_decimal()) { // decimal non-predicate column
+ for (; begin < end; begin++) {
+ const char* cur_ptr = (const char*)&_chunk.data[begin * SIZE_OF_TYPE];
+ int64_t int_value = *(int64_t*)(cur_ptr);
+ int32_t frac_value = *(int32_t*)(cur_ptr + sizeof(int64_t));
+ DecimalV2Value data(int_value, frac_value);
+ dst->insert_data(reinterpret_cast<char*>(&data), 0);
+ }
+ } else if (dst->is_date_type()) {
+ for (; begin < end; begin++) {
+ const char* cur_ptr = (const char*)&_chunk.data[begin * SIZE_OF_TYPE];
+ uint64_t value = 0;
+ value = *(unsigned char*)(cur_ptr + 2);
+ value <<= 8;
+ value |= *(unsigned char*)(cur_ptr + 1);
+ value <<= 8;
+ value |= *(unsigned char*)(cur_ptr);
+ DateTimeValue date;
+ date.from_olap_date(value);
+ dst->insert_data(reinterpret_cast<char*>(&date), 0);
+ }
+ } else {
+ // todo(wb) need performance test here, may be batch memory copy?
+ for (; begin < end; begin++) {
+ dst->insert_data((const char*)&_chunk.data[begin * SIZE_OF_TYPE], 0);
+ }
+ }
+
+ *n = max_fetch;
+ _cur_index += max_fetch;
+
+ return Status::OK();
+ };
+
Status peek_next_batch(size_t* n, ColumnBlockView* dst) override {
return next_batch<false>(n, dst);
}
@@ -393,6 +440,7 @@ private:
int _size_of_element;
size_t _cur_index;
Chunk _chunk;
+ friend class BinaryDictPageDecoder;
};
} // namespace segment_v2
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 3bbbeab..c08b328 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -30,6 +30,8 @@
#include "util/block_compression.h"
#include "util/coding.h" // for get_varint32
#include "util/rle_encoding.h" // for RleDecoder
+#include "vec/core/types.h"
+#include "vec/runtime/vdatetime_value.h" //for VecDateTime
namespace doris {
namespace segment_v2 {
@@ -578,6 +580,57 @@ Status FileColumnIterator::next_batch(size_t* n, ColumnBlockView* dst, bool* has
return Status::OK();
}
+Status FileColumnIterator::next_batch(size_t* n, vectorized::MutableColumnPtr &dst, bool* has_null) {
+ size_t curr_size = dst->byte_size();
+ size_t remaining = *n;
+ *has_null = false;
+ while (remaining > 0) {
+ if (!_page->has_remaining()) {
+ bool eos = false;
+ RETURN_IF_ERROR(_load_next_page(&eos));
+ if (eos) {
+ break;
+ }
+ }
+
+ // number of rows to be read from this page
+ size_t nrows_in_page = std::min(remaining, _page->remaining());
+ size_t nrows_to_read = nrows_in_page;
+ if (_page->has_null) {
+ while (nrows_to_read > 0) {
+ bool is_null = false;
+ size_t this_run = _page->null_decoder.GetNextRun(&is_null, nrows_to_read);
+ // we use num_rows only for CHECK
+ size_t num_rows = this_run;
+ if (!is_null) {
+ RETURN_IF_ERROR(_page->data_decoder->next_batch(&num_rows, dst));
+ DCHECK_EQ(this_run, num_rows);
+ } else {
+ *has_null = true;
+ // todo(wb) add a DCHECK here to check whether type is column nullable
+ for (size_t x = 0; x < this_run; x++) {
+ dst->insert_data(nullptr, 0); // todo(wb) vectorized here
+ }
+ }
+
+ nrows_to_read -= this_run;
+ _page->offset_in_page += this_run;
+ _current_ordinal += this_run;
+ }
+ } else {
+ RETURN_IF_ERROR(_page->data_decoder->next_batch(&nrows_to_read, dst));
+ DCHECK_EQ(nrows_to_read, nrows_in_page);
+
+ _page->offset_in_page += nrows_to_read;
+ _current_ordinal += nrows_to_read;
+ }
+ remaining -= nrows_in_page;
+ }
+ *n -= remaining;
+ _opts.stats->bytes_read += (dst->byte_size() - curr_size) + BitmapSize(*n);
+ return Status::OK();
+}
+
Status FileColumnIterator::_load_next_page(bool* eos) {
_page_iter.next();
if (!_page_iter.valid()) {
@@ -715,5 +768,55 @@ Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlockView* dst, b
return Status::OK();
}
+void DefaultValueColumnIterator::insert_default_data(vectorized::MutableColumnPtr &dst, size_t n) {
+ vectorized::Int128 int128;
+ char* data_ptr = (char*)&int128;
+ size_t data_len = sizeof(int128);
+
+ auto type = _type_info->type();
+ if (type == OLAP_FIELD_TYPE_DATE) {
+ assert(_type_size == sizeof(FieldTypeTraits<OLAP_FIELD_TYPE_DATE>::CppType)); //uint24_t
+ std::string str = FieldTypeTraits<OLAP_FIELD_TYPE_DATE>::to_string(_mem_value);
+
+ vectorized::VecDateTimeValue value;
+ value.from_date_str(str.c_str(), str.length());
+ value.cast_to_date();
+ //TODO: here is int128 = int64
+ int128 = binary_cast<vectorized::VecDateTimeValue, vectorized::Int64>(value);
+ } else if (type == OLAP_FIELD_TYPE_DATETIME) {
+ assert(_type_size == sizeof(FieldTypeTraits<OLAP_FIELD_TYPE_DATETIME>::CppType)); //int64_t
+ std::string str = FieldTypeTraits<OLAP_FIELD_TYPE_DATETIME>::to_string(_mem_value);
+
+ vectorized::VecDateTimeValue value;
+ value.from_date_str(str.c_str(), str.length());
+ value.to_datetime();
+
+ int128 = binary_cast<vectorized::VecDateTimeValue, vectorized::Int64>(value);
+ } else if (type == OLAP_FIELD_TYPE_DECIMAL) {
+ assert(_type_size == sizeof(FieldTypeTraits<OLAP_FIELD_TYPE_DECIMAL>::CppType)); //decimal12_t
+ decimal12_t* d = (decimal12_t*)_mem_value;
+ int128 = DecimalV2Value(d->integer, d->fraction).value();
+ } else {
+ data_ptr = (char*)_mem_value;
+ data_len = _type_size;
+ }
+
+ for (size_t i = 0; i < n; ++i) {
+ dst->insert_data(data_ptr, data_len);
+ }
+}
+
+Status DefaultValueColumnIterator::next_batch(size_t* n, vectorized::MutableColumnPtr &dst, bool* has_null) {
+ if (_is_default_value_null) {
+ *has_null = true;
+ dst->insert_many_defaults(*n);
+ } else {
+ *has_null = false;
+ insert_default_data(dst, *n);
+ }
+
+ return Status::OK();
+}
+
} // namespace segment_v2
} // namespace doris
diff --git a/be/src/olap/rowset/segment_v2/column_reader.h b/be/src/olap/rowset/segment_v2/column_reader.h
index 0b3588c..98afe44 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.h
+++ b/be/src/olap/rowset/segment_v2/column_reader.h
@@ -215,11 +215,20 @@ public:
return next_batch(n, dst, &has_null);
}
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) {
+ bool has_null;
+ return next_batch(n, dst, &has_null);
+ }
+
// After one seek, we can call this function many times to read data
// into ColumnBlockView. when read string type data, memory will allocated
// from MemPool
virtual Status next_batch(size_t* n, ColumnBlockView* dst, bool* has_null) = 0;
+ virtual Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst, bool* has_null) {
+ return Status::NotSupported("not implement");
+ }
+
virtual ordinal_t get_current_ordinal() const = 0;
virtual Status get_row_ranges_by_zone_map(CondColumn* cond_column, CondColumn* delete_condition,
@@ -268,6 +277,8 @@ public:
Status next_batch(size_t* n, ColumnBlockView* dst, bool* has_null) override;
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst, bool* has_null) override;
+
ordinal_t get_current_ordinal() const override { return _current_ordinal; }
// get row ranges by zone map
@@ -401,11 +412,20 @@ public:
return Status::OK();
}
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) {
+ bool has_null;
+ return next_batch(n, dst, &has_null);
+ }
+
Status next_batch(size_t* n, ColumnBlockView* dst, bool* has_null) override;
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst, bool* has_null) override;
+
ordinal_t get_current_ordinal() const override { return _current_rowid; }
private:
+ void insert_default_data(vectorized::MutableColumnPtr &dst, size_t n);
+
bool _has_default_value;
std::string _default_value;
bool _is_nullable;
diff --git a/be/src/olap/rowset/segment_v2/empty_segment_iterator.cpp b/be/src/olap/rowset/segment_v2/empty_segment_iterator.cpp
index 61c0fab..f367c2d 100644
--- a/be/src/olap/rowset/segment_v2/empty_segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/empty_segment_iterator.cpp
@@ -30,5 +30,9 @@ Status EmptySegmentIterator::next_batch(RowBlockV2* block) {
return Status::EndOfFile("no more data in segment");
}
+Status EmptySegmentIterator::next_batch(vectorized::Block* block) {
+ return Status::EndOfFile("no more data in segment");
+}
+
} // namespace segment_v2
-} // namespace doris
\ No newline at end of file
+} // namespace doris
diff --git a/be/src/olap/rowset/segment_v2/empty_segment_iterator.h b/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
index cb6c48c..3e1a4f9 100644
--- a/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
+++ b/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
@@ -32,10 +32,11 @@ public:
Status init(const StorageReadOptions& opts) override { return Status::OK(); }
const Schema& schema() const override { return _schema; }
Status next_batch(RowBlockV2* row_block) override;
+ Status next_batch(vectorized::Block* block) override;
private:
Schema _schema;
};
} // namespace segment_v2
-} // namespace doris
\ No newline at end of file
+} // namespace doris
diff --git a/be/src/olap/rowset/segment_v2/frame_of_reference_page.h b/be/src/olap/rowset/segment_v2/frame_of_reference_page.h
index 5d0c0b1..df8fe06 100644
--- a/be/src/olap/rowset/segment_v2/frame_of_reference_page.h
+++ b/be/src/olap/rowset/segment_v2/frame_of_reference_page.h
@@ -161,6 +161,10 @@ public:
return Status::OK();
}
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override {
+ return Status::NotSupported("frame page not implement vec op now");
+ };
+
Status peek_next_batch(size_t* n, ColumnBlockView* dst) override {
return next_batch<false>(n, dst);
}
diff --git a/be/src/olap/rowset/segment_v2/page_decoder.h b/be/src/olap/rowset/segment_v2/page_decoder.h
index feed37f..ab5b482 100644
--- a/be/src/olap/rowset/segment_v2/page_decoder.h
+++ b/be/src/olap/rowset/segment_v2/page_decoder.h
@@ -19,6 +19,7 @@
#include "common/status.h" // for Status
#include "olap/column_block.h" // for ColumnBlockView
+#include "vec/columns/column.h"
namespace doris {
namespace segment_v2 {
@@ -81,6 +82,10 @@ public:
// allocated in the column_vector_view's mem_pool.
virtual Status next_batch(size_t* n, ColumnBlockView* dst) = 0;
+ virtual Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) {
+ return Status::NotSupported("not implement vec op now");
+ }
+
// Same as `next_batch` except for not moving forward the cursor.
// When read array's ordinals in `ArrayFileColumnIterator`, we want to read one extra ordinal
// but do not want to move forward the cursor.
diff --git a/be/src/olap/rowset/segment_v2/plain_page.h b/be/src/olap/rowset/segment_v2/plain_page.h
index 555c9af..09e4280 100644
--- a/be/src/olap/rowset/segment_v2/plain_page.h
+++ b/be/src/olap/rowset/segment_v2/plain_page.h
@@ -186,6 +186,10 @@ public:
Status next_batch(size_t* n, ColumnBlockView* dst) override { return next_batch<true>(n, dst); }
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override {
+ return Status::NotSupported("plain page not implement vec op now");
+ };
+
template <bool forward_index>
inline Status next_batch(size_t* n, ColumnBlockView* dst) {
DCHECK(_parsed);
diff --git a/be/src/olap/rowset/segment_v2/rle_page.h b/be/src/olap/rowset/segment_v2/rle_page.h
index ed20aaf..52a9613 100644
--- a/be/src/olap/rowset/segment_v2/rle_page.h
+++ b/be/src/olap/rowset/segment_v2/rle_page.h
@@ -230,6 +230,29 @@ public:
return Status::OK();
}
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr &dst) override {
+ DCHECK(_parsed);
+ if (PREDICT_FALSE(*n == 0 || _cur_index >= _num_elements)) {
+ *n = 0;
+ return Status::OK();
+ }
+
+ size_t to_fetch = std::min(*n, static_cast<size_t>(_num_elements - _cur_index));
+ size_t remaining = to_fetch;
+ bool result = false;
+ CppType value;
+ while (remaining > 0) {
+ result = _rle_decoder.Get(&value);
+ DCHECK(result);
+ dst->insert_data((char*)(&value), SIZE_OF_TYPE);
+ remaining--;
+ }
+
+ _cur_index += to_fetch;
+ *n = to_fetch;
+ return Status::OK();
+ };
+
size_t count() const override { return _num_elements; }
size_t current_index() const override { return _cur_index; }
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index cb7e0d5..d19f56e 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -581,5 +581,10 @@ Status SegmentIterator::next_batch(RowBlockV2* block) {
return Status::OK();
}
+Status SegmentIterator::next_batch(vectorized::Block* block) {
+ //TODO
+ return Status::NotSupported("not implement now");
+}
+
} // namespace segment_v2
} // namespace doris
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.h b/be/src/olap/rowset/segment_v2/segment_iterator.h
index 0da6ae4..fabefd5 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.h
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.h
@@ -50,8 +50,11 @@ public:
SegmentIterator(std::shared_ptr<Segment> segment, const Schema& _schema,
std::shared_ptr<MemTracker> parent);
~SegmentIterator() override;
+
Status init(const StorageReadOptions& opts) override;
Status next_batch(RowBlockV2* row_block) override;
+ Status next_batch(vectorized::Block* block) override;
+
const Schema& schema() const override { return _schema; }
bool is_lazy_materialization_read() const override { return _lazy_materialization_read; }
uint64_t data_id() const { return _segment->id(); }
diff --git a/be/src/olap/schema.cpp b/be/src/olap/schema.cpp
index ab00e37..d588b58 100644
--- a/be/src/olap/schema.cpp
+++ b/be/src/olap/schema.cpp
@@ -103,4 +103,48 @@ Schema::~Schema() {
}
}
+vectorized::DataTypePtr Schema::get_data_type_ptr(FieldType type) {
+ switch (type) {
+ case OLAP_FIELD_TYPE_TINYINT:
+ return std::make_shared<vectorized::DataTypeInt8>();
+
+ case OLAP_FIELD_TYPE_SMALLINT:
+ return std::make_shared<vectorized::DataTypeInt16>();
+
+ case OLAP_FIELD_TYPE_INT:
+ return std::make_shared<vectorized::DataTypeInt32>();
+
+ case OLAP_FIELD_TYPE_FLOAT:
+ return std::make_shared<vectorized::DataTypeFloat32>();
+
+ case OLAP_FIELD_TYPE_BIGINT:
+ return std::make_shared<vectorized::DataTypeInt64>();
+
+ case OLAP_FIELD_TYPE_LARGEINT:
+ return std::make_shared<vectorized::DataTypeInt128>();
+
+ case OLAP_FIELD_TYPE_DATE:
+ return std::make_shared<vectorized::DataTypeDate>();
+
+ case OLAP_FIELD_TYPE_DATETIME:
+ return std::make_shared<vectorized::DataTypeDateTime>();
+
+ case OLAP_FIELD_TYPE_DOUBLE:
+ return std::make_shared<vectorized::DataTypeFloat64>();
+
+ case OLAP_FIELD_TYPE_CHAR:
+ case OLAP_FIELD_TYPE_VARCHAR:
+ case OLAP_FIELD_TYPE_HLL:
+ return std::make_shared<vectorized::DataTypeString>();
+
+ case OLAP_FIELD_TYPE_DECIMAL:
+ return std::make_shared<vectorized::DataTypeDecimal<vectorized::Decimal128>>(27, 9);
+
+ default:
+ DCHECK(false);
+ }
+ // For llvm complain
+ return nullptr;
+}
+
} // namespace doris
diff --git a/be/src/olap/schema.h b/be/src/olap/schema.h
index 39e3c79..3fd90bf 100644
--- a/be/src/olap/schema.h
+++ b/be/src/olap/schema.h
@@ -100,6 +100,10 @@ public:
~Schema();
+ static vectorized::DataTypePtr get_data_type_ptr(FieldType type);
+
+ const std::vector<Field*>& columns() const { return _cols; }
+
const Field* column(ColumnId cid) const { return _cols[cid]; }
Field* mutable_column(ColumnId cid) const { return _cols[cid]; }
diff --git a/be/src/olap/tablet_schema.cpp b/be/src/olap/tablet_schema.cpp
index 072a415..5b407d5 100644
--- a/be/src/olap/tablet_schema.cpp
+++ b/be/src/olap/tablet_schema.cpp
@@ -18,6 +18,8 @@
#include "olap/tablet_schema.h"
#include "tablet_meta.h"
+#include "vec/core/block.h"
+#include "vec/data_types/data_type.h"
namespace doris {
@@ -490,6 +492,16 @@ void TabletSchema::init_field_index_for_test() {
}
}
+vectorized::Block TabletSchema::create_block(const std::vector<uint32_t>& return_columns) const {
+ vectorized::Block block;
+ for (int i = 0; i < return_columns.size(); ++i) {
+ const auto& col = _cols[return_columns[i]];
+ auto data_type = vectorized::IDataType::from_olap_engine(col.type(), col.is_nullable());
+ block.insert({data_type->create_column(), data_type, col.name()});
+ }
+ return block;
+}
+
bool operator==(const TabletColumn& a, const TabletColumn& b) {
if (a._unique_id != b._unique_id) return false;
if (a._col_name != b._col_name) return false;
@@ -548,4 +560,5 @@ bool operator!=(const TabletSchema& a, const TabletSchema& b) {
return !(a == b);
}
+
} // namespace doris
diff --git a/be/src/olap/tablet_schema.h b/be/src/olap/tablet_schema.h
index 0cb894f..ffd8c34 100644
--- a/be/src/olap/tablet_schema.h
+++ b/be/src/olap/tablet_schema.h
@@ -26,6 +26,9 @@
#include "olap/types.h"
namespace doris {
+namespace vectorized {
+class Block;
+}
class TabletColumn {
public:
@@ -141,6 +144,7 @@ public:
inline void set_delete_sign_idx(int32_t delete_sign_idx) { _delete_sign_idx = delete_sign_idx; }
inline bool has_sequence_col() const { return _sequence_col_idx != -1; }
inline int32_t sequence_col_idx() const { return _sequence_col_idx; }
+ vectorized::Block create_block(const std::vector<uint32_t>& return_columns) const;
private:
// Only for unit test
diff --git a/be/src/olap/tuple_reader.cpp b/be/src/olap/tuple_reader.cpp
index 9eba1d2..5c15c2b 100644
--- a/be/src/olap/tuple_reader.cpp
+++ b/be/src/olap/tuple_reader.cpp
@@ -196,7 +196,7 @@ OLAPStatus TupleReader::_unique_key_next_row(RowCursor* row_cursor, MemPool* mem
cur_delete_flag = _next_delete_flag;
}
- // if reader needs to filter delete row and current delete_flag is ture,
+ // if reader needs to filter delete row and current delete_flag is true,
// then continue
if (!(cur_delete_flag && _filter_delete)) {
break;
diff --git a/be/src/runtime/datetime_value.cpp b/be/src/runtime/datetime_value.cpp
index 9ec8ccb..b545f55 100644
--- a/be/src/runtime/datetime_value.cpp
+++ b/be/src/runtime/datetime_value.cpp
@@ -1466,6 +1466,8 @@ bool DateTimeValue::from_date_format_str(const char* format, int format_len, con
}
bool DateTimeValue::date_add_interval(const TimeInterval& interval, TimeUnit unit) {
+ if (!is_valid_date()) return false;
+
int sign = interval.is_neg ? -1 : 1;
switch (unit) {
case MICROSECOND:
diff --git a/be/src/runtime/datetime_value.h b/be/src/runtime/datetime_value.h
index 096a24d..4f3a310 100644
--- a/be/src/runtime/datetime_value.h
+++ b/be/src/runtime/datetime_value.h
@@ -30,7 +30,7 @@
#include "udf/udf.h"
#include "util/hash_util.hpp"
#include "util/timezone_utils.h"
-
+#include "vec/runtime/vdatetime_value.h"
namespace doris {
enum TimeUnit {
@@ -566,7 +566,9 @@ public:
private:
// Used to make sure sizeof DateTimeValue
friend class UnusedClass;
-
+ friend void doris::vectorized::VecDateTimeValue::convert_vec_dt_to_dt(DateTimeValue* dt);
+ friend void doris::vectorized::VecDateTimeValue::convert_dt_to_vec_dt(DateTimeValue* dt);
+
void from_packed_time(int64_t packed_time) {
_microsecond = packed_time % (1LL << 24);
int64_t ymdhms = packed_time >> 24;
diff --git a/be/src/runtime/descriptors.cpp b/be/src/runtime/descriptors.cpp
index ab4fa97..95d0a79 100644
--- a/be/src/runtime/descriptors.cpp
+++ b/be/src/runtime/descriptors.cpp
@@ -25,6 +25,10 @@
#include "gen_cpp/Descriptors_types.h"
#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/columns_with_type_and_name.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/data_types/data_type_nullable.h"
+
namespace doris {
using boost::algorithm::join;
@@ -80,6 +84,21 @@ void SlotDescriptor::to_protobuf(PSlotDescriptor* pslot) const {
pslot->set_is_materialized(_is_materialized);
}
+vectorized::MutableColumnPtr SlotDescriptor::get_empty_mutable_column() const {
+ auto data_column = type().get_data_type_ptr()->create_column();
+ if (is_nullable()) {
+ return doris::vectorized::ColumnNullable::create(std::move(data_column), doris::vectorized::ColumnUInt8::create());
+ }
+ return data_column;
+}
+
+vectorized::DataTypePtr SlotDescriptor::get_data_type_ptr() const {
+ if (is_nullable()) {
+ return std::make_shared<vectorized::DataTypeNullable>(type().get_data_type_ptr());
+ }
+ return type().get_data_type_ptr();
+}
+
std::string SlotDescriptor::debug_string() const {
std::stringstream out;
out << "Slot(id=" << _id << " type=" << _type << " col=" << _col_pos
@@ -378,7 +397,7 @@ void RowDescriptor::to_thrift(std::vector<TTupleId>* row_tuple_ids) {
}
void RowDescriptor::to_protobuf(
- google::protobuf::RepeatedField<google::protobuf::int32>* row_tuple_ids) {
+ google::protobuf::RepeatedField<google::protobuf::int32>* row_tuple_ids) const {
row_tuple_ids->Clear();
for (auto desc : _tuple_desc_map) {
row_tuple_ids->Add(desc->id());
diff --git a/be/src/runtime/descriptors.h b/be/src/runtime/descriptors.h
index d668d42..97f9712 100644
--- a/be/src/runtime/descriptors.h
+++ b/be/src/runtime/descriptors.h
@@ -103,6 +103,9 @@ public:
std::string debug_string() const;
+ doris::vectorized::MutableColumnPtr get_empty_mutable_column() const;
+
+ doris::vectorized::DataTypePtr get_data_type_ptr() const;
private:
friend class DescriptorTbl;
friend class TupleDescriptor;
@@ -405,7 +408,7 @@ public:
// Populate row_tuple_ids with our ids.
void to_thrift(std::vector<TTupleId>* row_tuple_ids);
- void to_protobuf(google::protobuf::RepeatedField<google::protobuf::int32>* row_tuple_ids);
+ void to_protobuf(google::protobuf::RepeatedField<google::protobuf::int32>* row_tuple_ids) const;
// Return true if the tuple ids of this descriptor are a prefix
// of the tuple ids of other_desc.
diff --git a/be/src/runtime/exec_env.h b/be/src/runtime/exec_env.h
index 3cc2def..0b51e47 100644
--- a/be/src/runtime/exec_env.h
+++ b/be/src/runtime/exec_env.h
@@ -93,6 +93,7 @@ public:
const std::string& token() const;
ExternalScanContextMgr* external_scan_context_mgr() { return _external_scan_context_mgr; }
DataStreamMgr* stream_mgr() { return _stream_mgr; }
+ doris::vectorized::VDataStreamMgr* vstream_mgr() { return _vstream_mgr; }
ResultBufferMgr* result_mgr() { return _result_mgr; }
ResultQueueMgr* result_queue_mgr() { return _result_queue_mgr; }
ClientCache<BackendServiceClient>* client_cache() { return _backend_client_cache; }
@@ -163,6 +164,7 @@ private:
// Leave protected so that subclasses can override
ExternalScanContextMgr* _external_scan_context_mgr = nullptr;
DataStreamMgr* _stream_mgr = nullptr;
+ doris::vectorized::VDataStreamMgr* _vstream_mgr = nullptr;
ResultBufferMgr* _result_mgr = nullptr;
ResultQueueMgr* _result_queue_mgr = nullptr;
ClientCache<BackendServiceClient>* _backend_client_cache = nullptr;
diff --git a/be/src/runtime/exec_env_init.cpp b/be/src/runtime/exec_env_init.cpp
index a172bb6..35630d0 100644
--- a/be/src/runtime/exec_env_init.cpp
+++ b/be/src/runtime/exec_env_init.cpp
@@ -63,6 +63,7 @@
#include "util/parse_util.h"
#include "util/pretty_printer.h"
#include "util/priority_thread_pool.hpp"
+#include "vec/runtime/vdata_stream_mgr.h"
namespace doris {
@@ -85,6 +86,7 @@ Status ExecEnv::_init(const std::vector<StorePath>& store_paths) {
_store_paths = store_paths;
_external_scan_context_mgr = new ExternalScanContextMgr(this);
_stream_mgr = new DataStreamMgr();
+ _vstream_mgr = new doris::vectorized::VDataStreamMgr();
_result_mgr = new ResultBufferMgr();
_result_queue_mgr = new ResultQueueMgr();
_backend_client_cache = new BackendServiceClientCache(config::max_client_cache_size_per_host);
diff --git a/be/src/runtime/fold_constant_executor.cpp b/be/src/runtime/fold_constant_executor.cpp
index 94d3d9e..cd6d5ff 100644
--- a/be/src/runtime/fold_constant_executor.cpp
+++ b/be/src/runtime/fold_constant_executor.cpp
@@ -28,6 +28,9 @@
#include "common/object_pool.h"
#include "common/status.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vexpr_context.h"
+
#include "gen_cpp/internal_service.pb.h"
#include "gen_cpp/PaloInternalService_types.h"
@@ -96,6 +99,66 @@ Status FoldConstantExecutor::fold_constant_expr(
return Status::OK();
}
+Status FoldConstantExecutor::fold_constant_vexpr(
+ const TFoldConstantParams& params, PConstantExprResult* response) {
+ const auto& expr_map = params.expr_map;
+ auto expr_result_map = response->mutable_expr_result_map();
+
+ TQueryGlobals query_globals = params.query_globals;
+ // init
+ Status status = _init(query_globals);
+ if (UNLIKELY(!status.ok())) {
+ LOG(WARNING) << "Failed to init mem trackers, msg: " << status.get_error_msg();
+ return status;
+ }
+
+ for (const auto& m : expr_map) {
+ PExprResultMap pexpr_result_map;
+ for (const auto& n : m.second) {
+ vectorized::VExprContext* ctx = nullptr;
+ const TExpr& texpr = n.second;
+ // create expr tree from TExpr
+ RETURN_IF_ERROR(vectorized::VExpr::create_expr_tree(&_pool, texpr, &ctx));
+ // prepare and open context
+ status = _prepare_and_open(ctx);
+ if (UNLIKELY(!status.ok())) {
+ LOG(WARNING) << "Failed to init mem trackers, msg: " << status.get_error_msg();
+ return status;
+ }
+
+ vectorized::Block tmp_block;
+ int result_column = -1;
+ // calc vexpr
+ ctx->execute(&tmp_block, &result_column);
+ DCHECK(result_column != -1);
+ PrimitiveType root_type = ctx->root()->type().type;
+ // covert to thrift type
+ TPrimitiveType::type t_type = doris::to_thrift(root_type);
+
+ // collect result
+ PExprResult expr_result;
+ string result;
+ const auto& column_ptr = tmp_block.get_by_position(result_column).column;
+ if (column_ptr->is_nullable() && column_ptr->is_null_at(0)) {
+ expr_result.set_success(false);
+ } else {
+ expr_result.set_success(true);
+ result = _get_result<true>((void *) column_ptr->get_data_at(0).data, ctx->root()->type().type);
+ }
+
+ expr_result.set_content(std::move(result));
+ expr_result.mutable_type()->set_type(t_type);
+ pexpr_result_map.mutable_map()->insert({n.first, expr_result});
+
+ // close context expr
+ ctx->close(_runtime_state.get());
+ }
+ expr_result_map->insert({m.first, pexpr_result_map});
+ }
+
+ return Status::OK();
+}
+
Status FoldConstantExecutor::_init(const TQueryGlobals& query_globals) {
// init runtime state, runtime profile
TPlanFragmentExecParams params;
@@ -128,11 +191,13 @@ Status FoldConstantExecutor::_init(const TQueryGlobals& query_globals) {
return Status::OK();
}
-Status FoldConstantExecutor::_prepare_and_open(ExprContext* ctx) {
+template <typename Context>
+Status FoldConstantExecutor::_prepare_and_open(Context* ctx) {
ctx->prepare(_runtime_state.get(), RowDescriptor(), _mem_tracker);
return ctx->open(_runtime_state.get());
}
+template <bool is_vec>
string FoldConstantExecutor::_get_result(void* src, PrimitiveType slot_type){
switch (slot_type) {
case TYPE_BOOLEAN: {
@@ -176,10 +241,17 @@ string FoldConstantExecutor::_get_result(void* src, PrimitiveType slot_type){
}
case TYPE_DATE:
case TYPE_DATETIME: {
- const DateTimeValue date_value = *reinterpret_cast<DateTimeValue*>(src);
- char str[MAX_DTVALUE_STR_LEN];
- date_value.to_string(str);
- return str;
+ if constexpr (is_vec) {
+ auto date_value = reinterpret_cast<vectorized::VecDateTimeValue*>(src);
+ char str[MAX_DTVALUE_STR_LEN];
+ date_value->to_string(str);
+ return str;
+ } else {
+ const DateTimeValue date_value = *reinterpret_cast<DateTimeValue *>(src);
+ char str[MAX_DTVALUE_STR_LEN];
+ date_value.to_string(str);
+ return str;
+ }
}
case TYPE_DECIMALV2: {
return reinterpret_cast<DecimalV2Value*>(src)->to_string();
diff --git a/be/src/runtime/fold_constant_executor.h b/be/src/runtime/fold_constant_executor.h
index 93ee988..c7c5a38 100644
--- a/be/src/runtime/fold_constant_executor.h
+++ b/be/src/runtime/fold_constant_executor.h
@@ -36,12 +36,17 @@ class FoldConstantExecutor {
public:
// fold constant expr
Status fold_constant_expr(const TFoldConstantParams& params, PConstantExprResult* response);
+
+ // fold constant vexpr
+ Status fold_constant_vexpr(const TFoldConstantParams& params, PConstantExprResult* response);
private:
// init runtime_state and mem_tracker
Status _init(const TQueryGlobals& query_globals);
// prepare expr
- Status _prepare_and_open(ExprContext* ctx);
+ template <typename Context>
+ Status _prepare_and_open(Context* ctx);
+ template <bool is_vec = false>
std::string _get_result(void* src, PrimitiveType slot_type);
std::unique_ptr<RuntimeState> _runtime_state;
diff --git a/be/src/runtime/mysql_result_writer.cpp b/be/src/runtime/mysql_result_writer.cpp
index 1c2589c..eaf1bd7 100644
--- a/be/src/runtime/mysql_result_writer.cpp
+++ b/be/src/runtime/mysql_result_writer.cpp
@@ -29,6 +29,13 @@
#include "util/mysql_row_buffer.h"
#include "util/types.h"
+#include "vec/core/block.h"
+#include "vec/columns/column_vector.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/common/assert_cast.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vexpr_context.h"
+
namespace doris {
MysqlResultWriter::MysqlResultWriter(BufferControlBlock* sinker,
@@ -204,7 +211,6 @@ int MysqlResultWriter::_add_row_value(int index, const TypeDescriptor& type, voi
}
Status MysqlResultWriter::_add_one_row(TupleRow* row) {
- SCOPED_TIMER(_convert_tuple_timer);
_row_buffer->reset();
int num_columns = _output_expr_ctxs.size();
int buf_ret = 0;
diff --git a/be/src/runtime/mysql_result_writer.h b/be/src/runtime/mysql_result_writer.h
index 2754e0c..8b96fec 100644
--- a/be/src/runtime/mysql_result_writer.h
+++ b/be/src/runtime/mysql_result_writer.h
@@ -21,6 +21,8 @@
#include "runtime/result_writer.h"
#include "runtime/runtime_state.h"
+#include "vec/data_types/data_type.h"
+
namespace doris {
class TupleRow;
diff --git a/be/src/runtime/plan_fragment_executor.cpp b/be/src/runtime/plan_fragment_executor.cpp
index a464211..fe64a4f 100644
--- a/be/src/runtime/plan_fragment_executor.cpp
+++ b/be/src/runtime/plan_fragment_executor.cpp
@@ -42,6 +42,10 @@
#include "util/uid_util.h"
#include "util/logging.h"
+#include "vec/core/block.h"
+#include "vec/exec/vexchange_node.h"
+#include "vec/runtime/vdata_stream_mgr.h"
+
namespace doris {
PlanFragmentExecutor::PlanFragmentExecutor(ExecEnv* exec_env,
@@ -159,6 +163,7 @@ Status PlanFragmentExecutor::prepare(const TExecPlanFragmentParams& request,
int num_senders = find_with_default(params.per_exch_num_senders, exch_node->id(), 0);
DCHECK_GT(num_senders, 0);
if (_runtime_state->enable_vectorized_exec()) {
+ static_cast<doris::vectorized::VExchangeNode*>(exch_node)->set_num_senders(num_senders);
} else {
static_cast<ExchangeNode*>(exch_node)->set_num_senders(num_senders);
}
@@ -214,6 +219,7 @@ Status PlanFragmentExecutor::prepare(const TExecPlanFragmentParams& request,
_row_batch.reset(new RowBatch(_plan->row_desc(), _runtime_state->batch_size(),
_runtime_state->instance_mem_tracker().get()));
+ _block.reset(new doris::vectorized::Block());
// _row_batch->tuple_data_pool()->set_limits(*_runtime_state->mem_trackers());
VLOG_NOTICE << "plan_root=\n" << _plan->debug_string();
_prepared = true;
@@ -244,6 +250,7 @@ Status PlanFragmentExecutor::open() {
}
Status status = Status::OK();
if (_runtime_state->enable_vectorized_exec()) {
+ status = open_vectorized_internal();
} else {
status = open_internal();
}
@@ -259,6 +266,82 @@ Status PlanFragmentExecutor::open() {
return status;
}
+Status PlanFragmentExecutor::open_vectorized_internal() {
+ {
+ SCOPED_CPU_TIMER(_fragment_cpu_timer);
+ SCOPED_TIMER(profile()->total_time_counter());
+ RETURN_IF_ERROR(_plan->open(_runtime_state.get()));
+ }
+ if (_sink == nullptr) {
+ return Status::OK();
+ }
+ {
+ SCOPED_CPU_TIMER(_fragment_cpu_timer);
+ RETURN_IF_ERROR(_sink->open(runtime_state()));
+ }
+ doris::vectorized::Block* block = nullptr;
+ while (true) {
+ {
+ SCOPED_CPU_TIMER(_fragment_cpu_timer);
+ RETURN_IF_ERROR(get_vectorized_internal(&block));
+ }
+
+ if (block == NULL) {
+ break;
+ }
+
+ SCOPED_TIMER(profile()->total_time_counter());
+ SCOPED_CPU_TIMER(_fragment_cpu_timer);
+ // Collect this plan and sub plan statistics, and send to parent plan.
+ if (_collect_query_statistics_with_every_batch) {
+ _collect_query_statistics();
+ }
+ RETURN_IF_ERROR(_sink->send(runtime_state(), block));
+ }
+ {
+ SCOPED_TIMER(profile()->total_time_counter());
+ _collect_query_statistics();
+ Status status;
+ {
+ std::lock_guard<std::mutex> l(_status_lock);
+ status = _status;
+ }
+ status = _sink->close(runtime_state(), status);
+ RETURN_IF_ERROR(status);
+ }
+ // Setting to NULL ensures that the d'tor won't double-close the sink.
+ _sink.reset(nullptr);
+ _done = true;
+
+ stop_report_thread();
+ send_report(true);
+
+ return Status::OK();
+}
+Status PlanFragmentExecutor::get_vectorized_internal(::doris::vectorized::Block** block) {
+ if (_done) {
+ *block = nullptr;
+ return Status::OK();
+ }
+
+ auto vexec_node = static_cast<doris::ExecNode*>(_plan);
+ while (!_done) {
+ _block->clear_column_data(vexec_node->row_desc().num_materialized_slots());
+ SCOPED_TIMER(profile()->total_time_counter());
+ RETURN_IF_ERROR(vexec_node->get_next(_runtime_state.get(), _block.get(), &_done));
+
+ if (_block->rows() > 0) {
+ COUNTER_UPDATE(_rows_produced_counter, _block->rows());
+ *block = _block.get();
+ break;
+ }
+
+ *block = nullptr;
+ }
+
+ return Status::OK();
+}
+
Status PlanFragmentExecutor::open_internal() {
{
SCOPED_CPU_TIMER(_fragment_cpu_timer);
@@ -521,8 +604,14 @@ void PlanFragmentExecutor::cancel() {
.query_id(_query_id).instance_id(_runtime_state->fragment_instance_id());
DCHECK(_prepared);
_runtime_state->set_is_cancelled(true);
- _runtime_state->exec_env()->stream_mgr()->cancel(_runtime_state->fragment_instance_id());
- _runtime_state->exec_env()->result_mgr()->cancel(_runtime_state->fragment_instance_id());
+
+ // must close stream_mgr to avoid dead lock in Exchange Node
+ if (_runtime_state->enable_vectorized_exec()) {
+ _runtime_state->exec_env()->vstream_mgr()->cancel(_runtime_state->fragment_instance_id());
+ } else {
+ _runtime_state->exec_env()->stream_mgr()->cancel(_runtime_state->fragment_instance_id());
+ _runtime_state->exec_env()->result_mgr()->cancel(_runtime_state->fragment_instance_id());
+ }
}
void PlanFragmentExecutor::set_abort() {
diff --git a/be/src/runtime/plan_fragment_executor.h b/be/src/runtime/plan_fragment_executor.h
index 1fc4578..3cdb6bb 100644
--- a/be/src/runtime/plan_fragment_executor.h
+++ b/be/src/runtime/plan_fragment_executor.h
@@ -30,11 +30,11 @@
#include "runtime/runtime_state.h"
#include "util/hash_util.hpp"
#include "util/time.h"
+#include "vec/core/block.h"
namespace doris {
class QueryFragmentsCtx;
-class HdfsFsCache;
class ExecNode;
class RowDescriptor;
class RowBatch;
@@ -195,6 +195,7 @@ private:
// Created in prepare (if required), owned by this object.
std::unique_ptr<DataSink> _sink;
std::unique_ptr<RowBatch> _row_batch;
+ std::unique_ptr<doris::vectorized::Block> _block;
// Number of rows returned by this fragment
RuntimeProfile::Counter* _rows_produced_counter;
@@ -234,9 +235,11 @@ private:
// have been closed, a final report will have been sent and the report thread will
// have been stopped. _sink will be set to nullptr after successful execution.
Status open_internal();
+ Status open_vectorized_internal();
// Executes get_next() logic and returns resulting status.
Status get_next_internal(RowBatch** batch);
+ Status get_vectorized_internal(::doris::vectorized::Block** block);
// Stops report thread, if one is running. Blocks until report thread terminates.
// Idempotent.
diff --git a/be/src/runtime/primitive_type.h b/be/src/runtime/primitive_type.h
index 36113a5..224957f 100644
--- a/be/src/runtime/primitive_type.h
+++ b/be/src/runtime/primitive_type.h
@@ -27,6 +27,13 @@
#include "runtime/decimalv2_value.h"
#include "runtime/large_int_value.h"
#include "runtime/string_value.h"
+#include "udf/udf.h"
+
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/columns_number.h"
+#include "vec/core/types.h"
+#include "vec/runtime/vdatetime_value.h"
namespace doris {
@@ -60,6 +67,51 @@ enum PrimitiveType {
TYPE_STRING, /* 23 */
};
+inline PrimitiveType convert_type_to_primitive(FunctionContext::Type type) {
+ switch (type) {
+ case FunctionContext::Type::INVALID_TYPE:
+ return PrimitiveType::INVALID_TYPE;
+ case FunctionContext::Type::TYPE_DOUBLE:
+ return PrimitiveType::TYPE_DOUBLE;
+ case FunctionContext::Type::TYPE_NULL:
+ return PrimitiveType::TYPE_NULL;
+ case FunctionContext::Type::TYPE_CHAR:
+ return PrimitiveType::TYPE_CHAR;
+ case FunctionContext::Type::TYPE_VARCHAR:
+ return PrimitiveType::TYPE_VARCHAR;
+ case FunctionContext::Type::TYPE_STRING:
+ return PrimitiveType::TYPE_STRING;
+ case FunctionContext::Type::TYPE_DATETIME:
+ return PrimitiveType::TYPE_DATETIME;
+ case FunctionContext::Type::TYPE_DECIMALV2:
+ return PrimitiveType::TYPE_DECIMALV2;
+ case FunctionContext::Type::TYPE_BOOLEAN:
+ return PrimitiveType::TYPE_BOOLEAN;
+ case FunctionContext::Type::TYPE_ARRAY:
+ return PrimitiveType::TYPE_ARRAY;
+ case FunctionContext::Type::TYPE_OBJECT:
+ return PrimitiveType::TYPE_OBJECT;
+ case FunctionContext::Type::TYPE_HLL:
+ return PrimitiveType::TYPE_HLL;
+ case FunctionContext::Type::TYPE_TINYINT:
+ return PrimitiveType::TYPE_TINYINT;
+ case FunctionContext::Type::TYPE_SMALLINT:
+ return PrimitiveType::TYPE_SMALLINT;
+ case FunctionContext::Type::TYPE_INT:
+ return PrimitiveType::TYPE_INT;
+ case FunctionContext::Type::TYPE_BIGINT:
+ return PrimitiveType::TYPE_BIGINT;
+ case FunctionContext::Type::TYPE_LARGEINT:
+ return PrimitiveType::TYPE_LARGEINT;
+ case FunctionContext::Type::TYPE_DATE:
+ return PrimitiveType::TYPE_DATE;
+ default:
+ DCHECK(false);
+ }
+
+ return PrimitiveType::INVALID_TYPE;
+}
+
inline bool is_enumeration_type(PrimitiveType type) {
switch (type) {
case TYPE_FLOAT:
@@ -98,6 +150,10 @@ inline bool is_string_type(PrimitiveType type) {
return type == TYPE_CHAR || type == TYPE_VARCHAR || type == TYPE_STRING;
}
+inline bool has_variable_type(PrimitiveType type) {
+ return type == TYPE_CHAR || type == TYPE_VARCHAR || type == TYPE_OBJECT || type == TYPE_STRING;
+}
+
// Returns the byte size of 'type' Returns 0 for variable length types.
inline int get_byte_size(PrimitiveType type) {
switch (type) {
@@ -125,9 +181,9 @@ inline int get_byte_size(PrimitiveType type) {
case TYPE_DOUBLE:
return 8;
- case TYPE_LARGEINT:
case TYPE_DATETIME:
case TYPE_DATE:
+ case TYPE_LARGEINT:
case TYPE_DECIMALV2:
return 16;
@@ -220,63 +276,78 @@ struct PrimitiveTypeTraits {};
template <>
struct PrimitiveTypeTraits<TYPE_BOOLEAN> {
using CppType = bool;
+ using ColumnType = vectorized::ColumnUInt8;
};
template <>
struct PrimitiveTypeTraits<TYPE_TINYINT> {
using CppType = int8_t;
+ using ColumnType = vectorized::ColumnInt8;
};
template <>
struct PrimitiveTypeTraits<TYPE_SMALLINT> {
using CppType = int16_t;
+ using ColumnType = vectorized::ColumnInt16;
};
template <>
struct PrimitiveTypeTraits<TYPE_INT> {
using CppType = int32_t;
+ using ColumnType = vectorized::ColumnInt32;
};
template <>
struct PrimitiveTypeTraits<TYPE_BIGINT> {
using CppType = int64_t;
+ using ColumnType = vectorized::ColumnInt64;
};
template <>
struct PrimitiveTypeTraits<TYPE_FLOAT> {
using CppType = float;
+ using ColumnType = vectorized::ColumnFloat32;
};
template <>
struct PrimitiveTypeTraits<TYPE_TIME> {
using CppType = double;
+ using ColumnType = vectorized::ColumnFloat64;
};
template <>
struct PrimitiveTypeTraits<TYPE_DOUBLE> {
using CppType = double;
+ using ColumnType = vectorized::ColumnFloat64;
};
template <>
struct PrimitiveTypeTraits<TYPE_DATE> {
- using CppType = DateTimeValue;
+ using CppType = doris::DateTimeValue;
+ using ColumnType = vectorized::ColumnVector<vectorized::DateTime>;
};
template <>
struct PrimitiveTypeTraits<TYPE_DATETIME> {
- using CppType = DateTimeValue;
+ using CppType = doris::DateTimeValue;
+ using ColumnType = vectorized::ColumnVector<vectorized::DateTime>;
};
template <>
struct PrimitiveTypeTraits<TYPE_DECIMALV2> {
using CppType = DecimalV2Value;
+ using ColumnType = vectorized::ColumnDecimal<vectorized::Decimal128>;
};
template <>
struct PrimitiveTypeTraits<TYPE_LARGEINT> {
using CppType = __int128_t;
+ using ColumnType = vectorized::ColumnInt128;
};
template <>
struct PrimitiveTypeTraits<TYPE_CHAR> {
using CppType = StringValue;
+ using ColumnType = vectorized::ColumnString;
};
template <>
struct PrimitiveTypeTraits<TYPE_VARCHAR> {
using CppType = StringValue;
+ using ColumnType = vectorized::ColumnString;
};
template <>
struct PrimitiveTypeTraits<TYPE_STRING> {
using CppType = StringValue;
+ using ColumnType = vectorized::ColumnString;
};
} // namespace doris
diff --git a/be/src/runtime/raw_value.h b/be/src/runtime/raw_value.h
index e9e270e..e16bbf5 100644
--- a/be/src/runtime/raw_value.h
+++ b/be/src/runtime/raw_value.h
@@ -87,6 +87,9 @@ public:
// TODO: fix get_hash_value
static uint32_t zlib_crc32(const void* value, const TypeDescriptor& type, uint32_t seed);
+ // Same as the up function, only use in vec exec engine.
+ static uint32_t zlib_crc32(const void* value, size_t len, const TypeDescriptor& type, uint32_t seed);
+
// Compares both values.
// Return value is < 0 if v1 < v2, 0 if v1 == v2, > 0 if v1 > v2.
static int compare(const void* v1, const void* v2, const TypeDescriptor& type);
@@ -399,6 +402,59 @@ inline uint32_t RawValue::zlib_crc32(const void* v, const TypeDescriptor& type,
}
}
+// NOTE: this is just for split data, decimal use old doris hash function
+// Because crc32 hardware is not equal with zlib crc32
+inline uint32_t RawValue::zlib_crc32(const void* v, size_t len, const TypeDescriptor& type, uint32_t seed) {
+ // Hash_combine with v = 0
+ if (v == nullptr) {
+ uint32_t value = 0x9e3779b9;
+ return seed ^ (value + (seed << 6) + (seed >> 2));
+ }
+
+ switch (type.type) {
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ case TYPE_STRING:
+ case TYPE_CHAR: {
+ return HashUtil::zlib_crc_hash(v, len, seed);
+ }
+
+ case TYPE_BOOLEAN:
+ case TYPE_TINYINT:
+ return HashUtil::zlib_crc_hash(v, 1, seed);
+ case TYPE_SMALLINT:
+ return HashUtil::zlib_crc_hash(v, 2, seed);
+ case TYPE_INT:
+ return HashUtil::zlib_crc_hash(v, 4, seed);
+ case TYPE_BIGINT:
+ return HashUtil::zlib_crc_hash(v, 8, seed);
+ case TYPE_LARGEINT:
+ return HashUtil::zlib_crc_hash(v, 16, seed);
+ case TYPE_FLOAT:
+ return HashUtil::zlib_crc_hash(v, 4, seed);
+ case TYPE_DOUBLE:
+ return HashUtil::zlib_crc_hash(v, 8, seed);
+ case TYPE_DATE:
+ case TYPE_DATETIME: {
+ auto* date_val = (const vectorized::VecDateTimeValue*)v;
+ char buf[64];
+ int len = date_val->to_buffer(buf);
+ return HashUtil::zlib_crc_hash(buf, len, seed);
+ }
+
+ case TYPE_DECIMALV2: {
+ const DecimalV2Value* dec_val = (const DecimalV2Value*)v;
+ int64_t int_val = dec_val->int_value();
+ int32_t frac_val = dec_val->frac_value();
+ seed = HashUtil::zlib_crc_hash(&int_val, sizeof(int_val), seed);
+ return HashUtil::zlib_crc_hash(&frac_val, sizeof(frac_val), seed);
+ }
+ default:
+ DCHECK(false) << "invalid type: " << type;
+ return 0;
+ }
+}
+
} // namespace doris
#endif
diff --git a/be/src/runtime/result_sink.cpp b/be/src/runtime/result_sink.cpp
index 206b0f0..60538ca 100644
--- a/be/src/runtime/result_sink.cpp
+++ b/be/src/runtime/result_sink.cpp
@@ -29,6 +29,8 @@
#include "runtime/runtime_state.h"
#include "util/uid_util.h"
+#include "vec/exprs/vexpr.h"
+
namespace doris {
ResultSink::ResultSink(const RowDescriptor& row_desc, const std::vector<TExpr>& t_output_expr,
diff --git a/be/src/runtime/row_batch.cpp b/be/src/runtime/row_batch.cpp
index 85ba3c7..3e7914e 100644
--- a/be/src/runtime/row_batch.cpp
+++ b/be/src/runtime/row_batch.cpp
@@ -30,8 +30,8 @@
#include "runtime/string_value.h"
#include "runtime/tuple_row.h"
-//#include "vec/columns/column_vector.h"
-//#include "vec/core/block.h"
+#include "vec/columns/column_vector.h"
+#include "vec/core/block.h"
using std::vector;
@@ -625,6 +625,57 @@ void RowBatch::transfer_resource_ownership(RowBatch* dest) {
reset();
}
+vectorized::Block RowBatch::convert_to_vec_block() const {
+ std::vector<vectorized::MutableColumnPtr> columns;
+ for (const auto tuple_desc : _row_desc.tuple_descriptors()) {
+ for (const auto slot_desc : tuple_desc->slots()) {
+ columns.emplace_back(slot_desc->get_empty_mutable_column());
+ }
+ }
+
+ std::vector<SlotDescriptor*> slot_descs;
+ std::vector<int> tuple_idx;
+ int column_numbers = 0;
+ for (int i = 0; i < _row_desc.tuple_descriptors().size(); ++i) {
+ auto tuple_desc = _row_desc.tuple_descriptors()[i];
+ for (int j = 0; j < tuple_desc->slots().size(); ++j) {
+ slot_descs.push_back(tuple_desc->slots()[j]);
+ tuple_idx.push_back(i);
+ }
+ column_numbers += tuple_desc->slots().size();
+ }
+ for (int i = 0; i < column_numbers; ++i) {
+ auto slot_desc = slot_descs[i];
+ for (int j = 0; j < _num_rows; ++j) {
+ TupleRow* src_row = get_row(j);
+ auto tuple = src_row->get_tuple(tuple_idx[i]);
+ if (slot_desc->is_nullable() && tuple->is_null(slot_desc->null_indicator_offset())) {
+ columns[i]->insert_data(nullptr, 0);
+ } else if (slot_desc->type().is_string_type()) {
+ auto string_value =
+ static_cast<const StringValue*>(tuple->get_slot(slot_desc->tuple_offset()));
+ columns[i]->insert_data(string_value->ptr, string_value->len);
+ } else {
+ columns[i]->insert_data(
+ static_cast<const char*>(tuple->get_slot(slot_desc->tuple_offset())),
+ slot_desc->slot_size());
+ }
+ }
+ }
+
+ doris::vectorized::ColumnsWithTypeAndName columns_with_type_and_name;
+ auto n_columns = 0;
+ for (const auto tuple_desc : _row_desc.tuple_descriptors()) {
+ for (const auto slot_desc : tuple_desc->slots()) {
+ columns_with_type_and_name.emplace_back(columns[n_columns++]->get_ptr(),
+ slot_desc->get_data_type_ptr(),
+ slot_desc->col_name());
+ }
+ }
+
+ return {columns_with_type_and_name};
+}
+
size_t RowBatch::get_batch_size(const TRowBatch& batch) {
size_t result = batch.tuple_data.size();
result += batch.row_tuples.size() * sizeof(TTupleId);
@@ -688,6 +739,7 @@ void RowBatch::deep_copy_to(RowBatch* dst) {
}
dst->commit_rows(_num_rows);
}
+
// TODO: consider computing size of batches as they are built up
size_t RowBatch::total_byte_size() const {
size_t result = 0;
diff --git a/be/src/runtime/row_batch.h b/be/src/runtime/row_batch.h
index 596fa70..4333c73 100644
--- a/be/src/runtime/row_batch.h
+++ b/be/src/runtime/row_batch.h
@@ -43,6 +43,7 @@ class TupleRow;
class TupleDescriptor;
class PRowBatch;
+
// A RowBatch encapsulates a batch of rows, each composed of a number of tuples.
// The maximum number of rows is fixed at the time of construction, and the caller
// can add rows up to that capacity.
@@ -315,7 +316,7 @@ public:
// we firstly update dest resource, and then reset current resource
void transfer_resource_ownership(RowBatch* dest);
- void copy_row(const TupleRow* src, TupleRow* dest) const {
+ void copy_row(TupleRow* src, TupleRow* dest) {
memcpy(dest, src, _num_tuples_per_row * sizeof(Tuple*));
}
@@ -362,6 +363,8 @@ public:
static size_t get_batch_size(const TRowBatch& batch);
static size_t get_batch_size(const PRowBatch& batch);
+ vectorized::Block convert_to_vec_block() const;
+
int num_rows() const { return _num_rows; }
int capacity() const { return _capacity; }
diff --git a/be/src/runtime/string_value.hpp b/be/src/runtime/string_value.hpp
index dd29bff..c44115d 100644
--- a/be/src/runtime/string_value.hpp
+++ b/be/src/runtime/string_value.hpp
@@ -42,22 +42,19 @@ static inline int string_compare(const char* s1, int64_t n1, const char* s2, int
int64_t len) {
DCHECK_EQ(len, std::min(n1, n2));
#ifdef __SSE4_2__
- if (CpuInfo::is_supported(CpuInfo::SSE4_2)) {
- while (len >= sse_util::CHARS_PER_128_BIT_REGISTER) {
- __m128i xmm0 = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s1));
- __m128i xmm1 = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s2));
- int chars_match =
- _mm_cmpestri(xmm0, sse_util::CHARS_PER_128_BIT_REGISTER, xmm1,
- sse_util::CHARS_PER_128_BIT_REGISTER, sse_util::STRCMP_MODE);
- if (chars_match != sse_util::CHARS_PER_128_BIT_REGISTER) {
- return (unsigned char)s1[chars_match] - (unsigned char)s2[chars_match];
- }
- len -= sse_util::CHARS_PER_128_BIT_REGISTER;
- s1 += sse_util::CHARS_PER_128_BIT_REGISTER;
- s2 += sse_util::CHARS_PER_128_BIT_REGISTER;
+ while (len >= sse_util::CHARS_PER_128_BIT_REGISTER) {
+ __m128i xmm0 = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s1));
+ __m128i xmm1 = _mm_loadu_si128(reinterpret_cast<const __m128i*>(s2));
+ int chars_match =
+ _mm_cmpestri(xmm0, sse_util::CHARS_PER_128_BIT_REGISTER, xmm1,
+ sse_util::CHARS_PER_128_BIT_REGISTER, sse_util::STRCMP_MODE);
+ if (chars_match != sse_util::CHARS_PER_128_BIT_REGISTER) {
+ return (unsigned char)s1[chars_match] - (unsigned char)s2[chars_match];
}
+ len -= sse_util::CHARS_PER_128_BIT_REGISTER;
+ s1 += sse_util::CHARS_PER_128_BIT_REGISTER;
+ s2 += sse_util::CHARS_PER_128_BIT_REGISTER;
}
-
#endif
unsigned char u1, u2;
while (len-- > 0) {
diff --git a/be/src/runtime/types.h b/be/src/runtime/types.h
index e40bc5b..c2adca9 100644
--- a/be/src/runtime/types.h
+++ b/be/src/runtime/types.h
@@ -28,6 +28,12 @@
#include "runtime/collection_value.h"
#include "runtime/primitive_type.h"
#include "thrift/protocol/TDebugProtocol.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
namespace doris {
@@ -272,6 +278,58 @@ struct TypeDescriptor {
return -1;
}
+ inline doris::vectorized::DataTypePtr get_data_type_ptr() const {
+ switch (type) {
+ case TYPE_BOOLEAN:
+ return std::make_shared<vectorized::DataTypeUInt8>();
+
+ case TYPE_TINYINT:
+ return std::make_shared<vectorized::DataTypeInt8>();
+
+ case TYPE_SMALLINT:
+ return std::make_shared<vectorized::DataTypeInt16>();
+
+ case TYPE_INT:
+ return std::make_shared<vectorized::DataTypeInt32>();
+
+ case TYPE_FLOAT:
+ return std::make_shared<vectorized::DataTypeFloat32>();
+
+ case TYPE_BIGINT:
+ return std::make_shared<vectorized::DataTypeInt64>();
+
+ case TYPE_LARGEINT:
+ return std::make_shared<vectorized::DataTypeInt128>();
+ case TYPE_DATE:
+ return std::make_shared<vectorized::DataTypeDate>();
+ case TYPE_DATETIME:
+ return std::make_shared<vectorized::DataTypeDateTime>();
+ case TYPE_TIME:
+ case TYPE_DOUBLE:
+ return std::make_shared<vectorized::DataTypeFloat64>();
+
+ case TYPE_STRING:
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_HLL:
+ return std::make_shared<vectorized::DataTypeString>();
+ case TYPE_OBJECT:
+ return std::make_shared<vectorized::DataTypeBitMap>();
+
+ case TYPE_DECIMALV2:
+ return std::make_shared<vectorized::DataTypeDecimal<vectorized::Decimal128>>(27, 9);
+ // Just Mock A NULL Type in Vec Exec Engine
+ case TYPE_NULL:
+ return std::make_shared<vectorized::DataTypeUInt8>();
+
+ case INVALID_TYPE:
+ default:
+ DCHECK(false);
+ }
+ // For llvm complain
+ return nullptr;
+ }
+
static inline int get_decimal_byte_size(int precision) {
DCHECK_GT(precision, 0);
if (precision <= MAX_DECIMAL4_PRECISION) {
diff --git a/be/src/service/internal_service.cpp b/be/src/service/internal_service.cpp
index ee6cf7e..4106963 100644
--- a/be/src/service/internal_service.cpp
+++ b/be/src/service/internal_service.cpp
@@ -36,6 +36,7 @@
#include "util/string_util.h"
#include "util/thrift_util.h"
#include "util/uid_util.h"
+#include "vec/runtime/vdata_stream_mgr.h"
namespace doris {
@@ -422,7 +423,10 @@ Status PInternalServiceImpl<T>::_fold_constant_expr(const std::string& ser_reque
uint32_t len = ser_request.size();
RETURN_IF_ERROR(deserialize_thrift_msg(buf, &len, false, &t_request));
}
- return FoldConstantExecutor().fold_constant_expr(t_request, response);
+ if (!t_request.__isset.vec_exec || !t_request.vec_exec)
+ return FoldConstantExecutor().fold_constant_expr(t_request, response);
+
+ return FoldConstantExecutor().fold_constant_vexpr(t_request, response);
}
template <typename T>
@@ -432,6 +436,7 @@ void PInternalServiceImpl<T>::transmit_block(google::protobuf::RpcController* cn
google::protobuf::Closure* done) {
VLOG_ROW << "transmit data: fragment_instance_id=" << print_id(request->finst_id())
<< " node=" << request->node_id();
+ _exec_env->vstream_mgr()->transmit_block(request, &done);
if (done != nullptr) {
done->Run();
}
diff --git a/be/src/udf/udf.cpp b/be/src/udf/udf.cpp
index 0360f39..269bd88 100644
--- a/be/src/udf/udf.cpp
+++ b/be/src/udf/udf.cpp
@@ -131,6 +131,10 @@ void FunctionContextImpl::set_constant_args(const std::vector<doris_udf::AnyVal*
_constant_args = constant_args;
}
+void FunctionContextImpl::set_constant_cols(const std::vector<doris::ColumnPtrWrapper*>& constant_cols) {
+ _constant_cols = constant_cols;
+}
+
bool FunctionContextImpl::check_allocations_empty() {
if (_allocations.empty() && _external_bytes_tracked == 0) {
return true;
@@ -187,6 +191,7 @@ FunctionContext* FunctionContextImpl::clone(MemPool* pool) {
create_context(_state, pool, _intermediate_type, _return_type, _arg_types,
_varargs_buffer_size, _debug);
new_context->_impl->_constant_args = _constant_args;
+ new_context->_impl->_constant_cols = _constant_cols;
new_context->_impl->_fragment_local_fn_state = _fragment_local_fn_state;
return new_context;
}
diff --git a/be/src/udf/udf.h b/be/src/udf/udf.h
index 3b9ff5a..141219a 100644
--- a/be/src/udf/udf.h
+++ b/be/src/udf/udf.h
@@ -28,6 +28,7 @@
// object serves as the interface object between the UDF/UDA and the doris process.
namespace doris {
class FunctionContextImpl;
+class ColumnPtrWrapper;
struct StringValue;
struct BitmapValue;
struct DecimalV2Value;
@@ -224,12 +225,16 @@ public:
// FunctionContext* argument) is a constant (e.g. 5, "string", 1 + 1).
bool is_arg_constant(int arg_idx) const;
+ bool is_col_constant(int arg_idx) const;
+
// Returns a pointer to the value of the arg_idx-th input argument (0 indexed, not
// including the FunctionContext* argument). Returns nullptr if the argument is not
// constant. This function can be used to obtain user-specified constants in a UDF's
// Init() or Close() functions.
AnyVal* get_constant_arg(int arg_idx) const;
+ doris::ColumnPtrWrapper* get_constant_col(int arg_idx) const;
+
// Create a test FunctionContext object. The caller is responsible for calling delete
// on it. This context has additional debugging validation enabled.
static FunctionContext* create_test_context();
diff --git a/be/src/udf/udf_internal.h b/be/src/udf/udf_internal.h
index 2d2c318..085002d 100644
--- a/be/src/udf/udf_internal.h
+++ b/be/src/udf/udf_internal.h
@@ -32,6 +32,7 @@ namespace doris {
class FreePool;
class MemPool;
class RuntimeState;
+class ColumnPtrWrapper;
// This class actually implements the interface of FunctionContext. This is split to
// hide the details from the external header.
@@ -67,6 +68,8 @@ public:
void set_constant_args(const std::vector<doris_udf::AnyVal*>& constant_args);
+ void set_constant_cols(const std::vector<doris::ColumnPtrWrapper*>& cols);
+
uint8_t* varargs_buffer() { return _varargs_buffer; }
std::vector<doris_udf::AnyVal*>* staging_input_vals() { return &_staging_input_vals; }
@@ -169,6 +172,8 @@ private:
// value of the argument.
std::vector<doris_udf::AnyVal*> _constant_args;
+ std::vector<doris::ColumnPtrWrapper*> _constant_cols;
+
// Used by ScalarFnCall to store the arguments when running without codegen. Allows us
// to pass AnyVal* arguments to the scalar function directly, rather than codegening a
// call that passes the correct AnyVal subclass pointer type.
diff --git a/be/src/udf/udf_ir.cpp b/be/src/udf/udf_ir.cpp
index 4eefd17..a1ceb02 100644
--- a/be/src/udf/udf_ir.cpp
+++ b/be/src/udf/udf_ir.cpp
@@ -25,6 +25,13 @@ bool FunctionContext::is_arg_constant(int i) const {
return _impl->_constant_args[i] != nullptr;
}
+bool FunctionContext::is_col_constant(int i) const {
+ if (i < 0 || i >= _impl->_constant_cols.size()) {
+ return false;
+ }
+ return _impl->_constant_cols[i] != nullptr;
+}
+
AnyVal* FunctionContext::get_constant_arg(int i) const {
if (i < 0 || i >= _impl->_constant_args.size()) {
return nullptr;
@@ -32,6 +39,13 @@ AnyVal* FunctionContext::get_constant_arg(int i) const {
return _impl->_constant_args[i];
}
+doris::ColumnPtrWrapper* FunctionContext::get_constant_col(int i) const {
+ if (i < 0 || i >= _impl->_constant_cols.size()) {
+ return nullptr;
+ }
+ return _impl->_constant_cols[i];
+}
+
int FunctionContext::get_num_args() const {
return _impl->_arg_types.size();
}
diff --git a/be/src/util/binary_cast.hpp b/be/src/util/binary_cast.hpp
index d528d71..764f5b5 100644
--- a/be/src/util/binary_cast.hpp
+++ b/be/src/util/binary_cast.hpp
@@ -23,7 +23,7 @@
#include "runtime/datetime_value.h"
#include "runtime/decimalv2_value.h"
#include "util/types.h"
-
+#include "vec/runtime/vdatetime_value.h"
namespace doris {
union TypeConverter {
uint64_t u64;
@@ -56,6 +56,11 @@ union DateTimeInt128Union {
~DateTimeInt128Union() {}
};
+union VecDateTimeInt64Union {
+ doris::vectorized::VecDateTimeValue dt;
+ __int64_t i64;
+ ~VecDateTimeInt64Union() {}
+};
// similar to reinterpret_cast but won't break strict-aliasing rules
template <typename From, typename To>
To binary_cast(From from) {
@@ -66,11 +71,13 @@ To binary_cast(From from) {
constexpr bool from_decv2_to_packed128 = match_v<From, DecimalV2Value, To, PackedInt128>;
constexpr bool from_i128_to_dt = match_v<From, __int128_t, To, DateTimeValue>;
constexpr bool from_dt_to_i128 = match_v<From, DateTimeValue, To, __int128_t>;
+ constexpr bool from_i64_to_vec_dt = match_v<From, __int64_t, To, doris::vectorized::VecDateTimeValue>;
+ constexpr bool from_vec_dt_to_i64 = match_v<From, doris::vectorized::VecDateTimeValue, To, __int64_t>;
constexpr bool from_i128_to_decv2 = match_v<From, __int128_t, To, DecimalV2Value>;
constexpr bool from_decv2_to_i128 = match_v<From, DecimalV2Value, To, __int128_t>;
static_assert(from_u64_to_db || from_i64_to_db || from_db_to_i64 || from_db_to_u64 ||
- from_decv2_to_packed128 || from_i128_to_dt || from_dt_to_i128 ||
+ from_decv2_to_packed128 || from_i128_to_dt || from_dt_to_i128 || from_i64_to_vec_dt || from_vec_dt_to_i64 ||
from_i128_to_decv2 || from_decv2_to_i128);
if constexpr (from_u64_to_db) {
@@ -99,6 +106,12 @@ To binary_cast(From from) {
} else if constexpr (from_dt_to_i128) {
DateTimeInt128Union conv = {.dt = from};
return conv.i128;
+ } else if constexpr (from_i64_to_vec_dt) {
+ VecDateTimeInt64Union conv = {.i64 = from};
+ return conv.dt;
+ } else if constexpr (from_vec_dt_to_i64) {
+ VecDateTimeInt64Union conv = {.dt = from};
+ return conv.i64;
} else if constexpr (from_i128_to_decv2) {
DecimalInt128Union conv;
conv.i128 = from;
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index 81fb10b..bdb124b 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -1691,6 +1691,12 @@ public:
}
return count;
}
+
+ void clear() {
+ _type = EMPTY;
+ _bitmap.clear();
+ _sv = 0;
+ }
// Implement an iterator for convenience
friend class BitmapValueIterator;
diff --git a/be/src/util/brpc_stub_cache.h b/be/src/util/brpc_stub_cache.h
index 53ee3b7..e944aac 100644
--- a/be/src/util/brpc_stub_cache.h
+++ b/be/src/util/brpc_stub_cache.h
@@ -66,7 +66,7 @@ public:
return stub;
}
- inline std::shared_ptr<PBackendService_Stub> get_stub(const TNetworkAddress& taddr) {
+ virtual std::shared_ptr<PBackendService_Stub> get_stub(const TNetworkAddress& taddr) {
butil::EndPoint endpoint;
if (str2endpoint(taddr.hostname.c_str(), taddr.port, &endpoint)) {
LOG(WARNING) << "unknown endpoint, hostname=" << taddr.hostname
diff --git a/be/src/util/runtime_profile.h b/be/src/util/runtime_profile.h
index f2632d5..1e9366f 100644
--- a/be/src/util/runtime_profile.h
+++ b/be/src/util/runtime_profile.h
@@ -52,8 +52,9 @@ namespace doris {
ScopedTimer<ThreadCpuStopWatch> MACRO_CONCAT(SCOPED_TIMER, __COUNTER__)(c)
#define CANCEL_SAFE_SCOPED_TIMER(c, is_cancelled) \
ScopedTimer<MonotonicStopWatch> MACRO_CONCAT(SCOPED_TIMER, __COUNTER__)(c, is_cancelled)
-#define SCOPED_RAW_TIMER(c) \
- ScopedRawTimer<MonotonicStopWatch, int64_t> MACRO_CONCAT(SCOPED_RAW_TIMER, __COUNTER__)(c)
+#define SCOPED_RAW_TIMER(c) \
+ doris::ScopedRawTimer<doris::MonotonicStopWatch, int64_t> MACRO_CONCAT(SCOPED_RAW_TIMER, \
+ __COUNTER__)(c)
#define SCOPED_ATOMIC_TIMER(c) \
ScopedRawTimer<MonotonicStopWatch, std::atomic<int64_t>> MACRO_CONCAT(SCOPED_ATOMIC_TIMER, \
__COUNTER__)(c)
diff --git a/be/src/util/static_asserts.cpp b/be/src/util/static_asserts.cpp
index eb50636..43833ad 100644
--- a/be/src/util/static_asserts.cpp
+++ b/be/src/util/static_asserts.cpp
@@ -17,7 +17,7 @@
#include "runtime/datetime_value.h"
#include "runtime/string_value.h"
-
+#include "vec/runtime/vdatetime_value.h"
namespace doris {
// This class is unused. It contains static (compile time) asserts.
// This is useful to validate struct sizes and other similar things
@@ -28,6 +28,7 @@ private:
static_assert(offsetof(StringValue, len) == 8);
// Datetime value
static_assert(sizeof(DateTimeValue) == 16);
+ static_assert(sizeof(doris::vectorized::VecDateTimeValue) == 8);
// static_assert(offsetof(DateTimeValue, _year) == 8);
};
diff --git a/be/src/util/url_coding.cpp b/be/src/util/url_coding.cpp
index 862c34cc..3c41114 100644
--- a/be/src/util/url_coding.cpp
+++ b/be/src/util/url_coding.cpp
@@ -194,6 +194,10 @@ int64_t base64_decode(const char* data, size_t length, char* decoded_data) {
// run through the whole string, converting as we go
while ((ch = *current++) != '\0' && length-- > 0) {
+ if (ch >= 256 || ch < 0) {
+ return -1;
+ }
+
if (ch == base64_pad) {
if (*current != '=' && (i % 4) == 1) {
return -1;
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
new file mode 100644
index 0000000..1738819
--- /dev/null
+++ b/be/src/vec/CMakeLists.txt
@@ -0,0 +1,153 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# where to put generated libraries
+set(LIBRARY_OUTPUT_PATH "${BUILD_DIR}/src/vec")
+# where to put generated binaries
+set(EXECUTABLE_OUTPUT_PATH "${BUILD_DIR}/src/vec")
+
+set(VEC_FILES
+ aggregate_functions/aggregate_function_avg.cpp
+ aggregate_functions/aggregate_function_count.cpp
+ aggregate_functions/aggregate_function_distinct.cpp
+ aggregate_functions/aggregate_function_sum.cpp
+ aggregate_functions/aggregate_function_min_max.cpp
+ aggregate_functions/aggregate_function_null.cpp
+ aggregate_functions/aggregate_function_uniq.cpp
+ aggregate_functions/aggregate_function_hll_union_agg.cpp
+ aggregate_functions/aggregate_function_bitmap.cpp
+ aggregate_functions/aggregate_function_reader.cpp
+ aggregate_functions/aggregate_function_window.cpp
+ aggregate_functions/aggregate_function_simple_factory.cpp
+ columns/collator.cpp
+ columns/column.cpp
+ columns/column_const.cpp
+ columns/column_decimal.cpp
+ columns/column_nullable.cpp
+ columns/column_string.cpp
+ columns/column_vector.cpp
+ columns/columns_common.cpp
+ common/demangle.cpp
+ common/exception.cpp
+ common/pod_array.cpp
+ common/string_utils/string_utils.cpp
+ core/block.cpp
+ core/block_info.cpp
+ core/column_with_type_and_name.cpp
+ core/field.cpp
+ core/field.cpp
+ core/sort_block.cpp
+ core/materialize_block.cpp
+ data_types/data_type.cpp
+ data_types/data_type_bitmap.cpp
+ data_types/data_type_nothing.cpp
+ data_types/data_type_nothing.cpp
+ data_types/data_type_nullable.cpp
+ data_types/data_type_nullable.cpp
+ data_types/data_type_number_base.cpp
+ data_types/data_type_string.cpp
+ data_types/data_type_decimal.cpp
+ data_types/get_least_supertype.cpp
+ data_types/nested_utils.cpp
+ data_types/data_type_date.cpp
+ data_types/data_type_date_time.cpp
+ exec/vaggregation_node.cpp
+ exec/ves_http_scan_node.cpp
+ exec/ves_http_scanner.cpp
+ exec/volap_scan_node.cpp
+ exec/vsort_node.cpp
+ exec/vsort_exec_exprs.cpp
+ exec/volap_scanner.cpp
+ exec/vexchange_node.cpp
+ exec/vset_operation_node.cpp
+ exec/vunion_node.cpp
+ exec/vintersect_node.cpp
+ exec/vexcept_node.cpp
+ exec/vselect_node.cpp
+ exec/vblocking_join_node.cpp
+ exec/vcross_join_node.cpp
+ exec/vodbc_scan_node.cpp
+ exec/vmysql_scan_node.cpp
+ exec/vschema_scan_node.cpp
+ exec/vempty_set_node.cpp
+ exec/vanalytic_eval_node.cpp
+ exec/vassert_num_rows_node.cpp
+ exec/join/vhash_join_node.cpp
+ exprs/vectorized_agg_fn.cpp
+ exprs/vectorized_fn_call.cpp
+ exprs/vexpr.cpp
+ exprs/vexpr_context.cpp
+ exprs/vliteral.cpp
+ exprs/vin_predicate.cpp
+ exprs/vslot_ref.cpp
+ exprs/vcast_expr.cpp
+ exprs/vcase_expr.cpp
+ exprs/vinfo_func.cpp
+ functions/math.cpp
+ functions/function_bitmap.cpp
+ functions/comparison.cpp
+ functions/comparison_less.cpp
+ functions/comparison_equals.cpp
+ functions/comparison_greater.cpp
+ functions/function.cpp
+ functions/function_helpers.cpp
+ functions/function_hash.cpp
+ functions/functions_logical.cpp
+ functions/function_case.cpp
+ functions/function_cast.cpp
+ functions/function_string.cpp
+ functions/function_timestamp.cpp
+ functions/comparison_equal_for_null.cpp
+ functions/function_json.cpp
+ functions/hll_cardinality.cpp
+ functions/hll_empty.cpp
+ functions/hll_hash.cpp
+ functions/plus.cpp
+ functions/modulo.cpp
+ functions/multiply.cpp
+ functions/minus.cpp
+ functions/int_div.cpp
+ functions/divide.cpp
+ functions/function_bit.cpp
+ functions/is_null.cpp
+ functions/is_not_null.cpp
+ functions/in.cpp
+ functions/like.cpp
+ functions/to_time_function.cpp
+ functions/time_of_function.cpp
+ functions/if.cpp
+ functions/function_ifnull.cpp
+ functions/nullif.cpp
+ functions/random.cpp
+ functions/function_date_or_datetime_computation.cpp
+ functions/function_date_or_datetime_to_string.cpp
+ functions/function_datetime_string_to_string.cpp
+ olap/vgeneric_iterators.cpp
+ olap/vcollect_iterator.cpp
+ olap/block_reader.cpp
+ sink/mysql_result_writer.cpp
+ sink/result_sink.cpp
+ sink/vdata_stream_sender.cpp
+ sink/vtabet_sink.cpp
+ runtime/vdatetime_value.cpp
+ runtime/vdata_stream_recvr.cpp
+ runtime/vdata_stream_mgr.cpp
+ runtime/vpartition_info.cpp
+ runtime/vsorted_run_merger.cpp)
+
+add_library(Vec STATIC
+ ${VEC_FILES}
+)
diff --git a/be/src/vec/aggregate_functions/aggregate_function.h b/be/src/vec/aggregate_functions/aggregate_function.h
new file mode 100644
index 0000000..412382f
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function.h
@@ -0,0 +1,237 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/IAggregateFunction.h
+// and modified by Doris
+
+#pragma once
+
+#include <cstddef>
+#include <istream>
+#include <memory>
+#include <ostream>
+#include <type_traits>
+#include <vector>
+
+#include "vec/common/exception.h"
+#include "vec/core/block.h"
+#include "vec/core/column_numbers.h"
+#include "vec/core/field.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+class Arena;
+class IColumn;
+class IDataType;
+
+using DataTypePtr = std::shared_ptr<const IDataType>;
+using DataTypes = std::vector<DataTypePtr>;
+
+using AggregateDataPtr = char*;
+using ConstAggregateDataPtr = const char*;
+
+/** Aggregate functions interface.
+ * Instances of classes with this interface do not contain the data itself for aggregation,
+ * but contain only metadata (description) of the aggregate function,
+ * as well as methods for creating, deleting and working with data.
+ * The data resulting from the aggregation (intermediate computing states) is stored in other objects
+ * (which can be created in some memory pool),
+ * and IAggregateFunction is the external interface for manipulating them.
+ */
+class IAggregateFunction {
+public:
+ IAggregateFunction(const DataTypes& argument_types_, const Array& parameters_)
+ : argument_types(argument_types_), parameters(parameters_) {}
+
+ /// Get main function name.
+ virtual String get_name() const = 0;
+
+ /// Get the result type.
+ virtual DataTypePtr get_return_type() const = 0;
+
+ virtual ~IAggregateFunction() {}
+
+ /** Create empty data for aggregation with `placement new` at the specified location.
+ * You will have to destroy them using the `destroy` method.
+ */
+ virtual void create(AggregateDataPtr __restrict place) const = 0;
+
+ /// Delete data for aggregation.
+ virtual void destroy(AggregateDataPtr __restrict place) const noexcept = 0;
+
+ /// Reset aggregation state
+ virtual void reset(AggregateDataPtr place) const = 0;
+
+ /// It is not necessary to delete data.
+ virtual bool has_trivial_destructor() const = 0;
+
+ /// Get `sizeof` of structure with data.
+ virtual size_t size_of_data() const = 0;
+
+ /// How the data structure should be aligned. NOTE: Currently not used (structures with aggregation state are put without alignment).
+ virtual size_t align_of_data() const = 0;
+
+ /** Adds a value into aggregation data on which place points to.
+ * columns points to columns containing arguments of aggregation function.
+ * row_num is number of row which should be added.
+ * Additional parameter arena should be used instead of standard memory allocator if the addition requires memory allocation.
+ */
+ virtual void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena* arena) const = 0;
+
+ /// Merges state (on which place points to) with other state of current aggregation function.
+ virtual void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena* arena) const = 0;
+
+ /// Serializes state (to transmit it over the network, for example).
+ virtual void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const = 0;
+
+ /// Deserializes state. This function is called only for empty (just created) states.
+ virtual void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena* arena) const = 0;
+
+ /// Returns true if a function requires Arena to handle own states (see add(), merge(), deserialize()).
+ virtual bool allocates_memory_in_arena() const { return false; }
+
+ /// Inserts results into a column.
+ virtual void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const = 0;
+
+ /** Returns true for aggregate functions of type -State.
+ * They are executed as other aggregate functions, but not finalized (return an aggregation state that can be combined with another).
+ */
+ virtual bool is_state() const { return false; }
+
+ /** The inner loop that uses the function pointer is better than using the virtual function.
+ * The reason is that in the case of virtual functions GCC 5.1.2 generates code,
+ * which, at each iteration of the loop, reloads the function address (the offset value in the virtual function table) from memory to the register.
+ * This gives a performance drop on simple queries around 12%.
+ * After the appearance of better compilers, the code can be removed.
+ */
+ using AddFunc = void (*)(const IAggregateFunction*, AggregateDataPtr, const IColumn**, size_t,
+ Arena*);
+ virtual AddFunc get_address_of_add_function() const = 0;
+
+ /** Contains a loop with calls to "add" function. You can collect arguments into array "places"
+ * and do a single call to "add_batch" for devirtualization and inlining.
+ */
+ virtual void add_batch(size_t batch_size, AggregateDataPtr* places, size_t place_offset,
+ const IColumn** columns, Arena* arena) const = 0;
+
+ /** The same for single place.
+ */
+ virtual void add_batch_single_place(size_t batch_size, AggregateDataPtr place,
+ const IColumn** columns, Arena* arena) const = 0;
+
+ // only used at agg reader
+ virtual void add_batch_range(size_t batch_begin, size_t batch_end, AggregateDataPtr place,
+ const IColumn** columns, Arena* arena, bool has_null = false) = 0;
+
+ // only used at window function
+ virtual void add_range_single_place(int64_t partition_start, int64_t partition_end,
+ int64_t frame_start, int64_t frame_end,
+ AggregateDataPtr place, const IColumn** columns,
+ Arena* arena) const = 0;
+
+ /** This is used for runtime code generation to determine, which header files to include in generated source.
+ * Always implement it as
+ * const char * get_header_file_path() const override { return __FILE__; }
+ */
+ virtual const char* get_header_file_path() const = 0;
+
+ const DataTypes& get_argument_types() const { return argument_types; }
+ const Array& get_parameters() const { return parameters; }
+
+protected:
+ DataTypes argument_types;
+ Array parameters;
+};
+
+/// Implement method to obtain an address of 'add' function.
+template <typename Derived>
+class IAggregateFunctionHelper : public IAggregateFunction {
+private:
+ static void add_free(const IAggregateFunction* that, AggregateDataPtr place,
+ const IColumn** columns, size_t row_num, Arena* arena) {
+ static_cast<const Derived&>(*that).add(place, columns, row_num, arena);
+ }
+
+public:
+ IAggregateFunctionHelper(const DataTypes& argument_types_, const Array& parameters_)
+ : IAggregateFunction(argument_types_, parameters_) {}
+
+ AddFunc get_address_of_add_function() const override { return &add_free; }
+
+ void add_batch(size_t batch_size, AggregateDataPtr* places, size_t place_offset,
+ const IColumn** columns, Arena* arena) const override {
+ for (size_t i = 0; i < batch_size; ++i)
+ static_cast<const Derived*>(this)->add(places[i] + place_offset, columns, i, arena);
+ }
+
+ void add_batch_single_place(size_t batch_size, AggregateDataPtr place, const IColumn** columns,
+ Arena* arena) const override {
+ for (size_t i = 0; i < batch_size; ++i)
+ static_cast<const Derived*>(this)->add(place, columns, i, arena);
+ }
+ //now this is use for sum/count/avg/min/max win function, other win function should override this function in class
+ void add_range_single_place(int64_t partition_start, int64_t partition_end, int64_t frame_start,
+ int64_t frame_end, AggregateDataPtr place, const IColumn** columns,
+ Arena* arena) const override {
+ frame_start = std::max<int64_t>(frame_start, partition_start);
+ frame_end = std::min<int64_t>(frame_end, partition_end);
+ for (int64_t i = frame_start; i < frame_end; ++i) {
+ static_cast<const Derived*>(this)->add(place, columns, i, arena);
+ }
+ }
+
+ void add_batch_range(size_t batch_begin, size_t batch_end, AggregateDataPtr place,
+ const IColumn** columns, Arena* arena, bool has_null) override {
+ for (size_t i = batch_begin; i <= batch_end; ++i)
+ static_cast<const Derived*>(this)->add(place, columns, i, arena);
+ }
+};
+
+/// Implements several methods for manipulation with data. T - type of structure with data for aggregation.
+template <typename T, typename Derived>
+class IAggregateFunctionDataHelper : public IAggregateFunctionHelper<Derived> {
+protected:
+ using Data = T;
+
+ static Data& data(AggregateDataPtr __restrict place) { return *reinterpret_cast<Data*>(place); }
+ static const Data& data(ConstAggregateDataPtr __restrict place) {
+ return *reinterpret_cast<const Data*>(place);
+ }
+
+public:
+ IAggregateFunctionDataHelper(const DataTypes& argument_types_, const Array& parameters_)
+ : IAggregateFunctionHelper<Derived>(argument_types_, parameters_) {}
+
+ void create(AggregateDataPtr __restrict place) const override { new (place) Data; }
+
+ void destroy(AggregateDataPtr __restrict place) const noexcept override { data(place).~Data(); }
+
+ bool has_trivial_destructor() const override { return std::is_trivially_destructible_v<Data>; }
+
+ size_t size_of_data() const override { return sizeof(Data); }
+
+ /// NOTE: Currently not used (structures with aggregation state are put without alignment).
+ size_t align_of_data() const override { return alignof(Data); }
+
+ void reset(AggregateDataPtr place) const override {}
+};
+
+using AggregateFunctionPtr = std::shared_ptr<IAggregateFunction>;
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_avg.cpp b/be/src/vec/aggregate_functions/aggregate_function_avg.cpp
new file mode 100644
index 0000000..bb7605b
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_avg.cpp
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionAvg.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_avg.h"
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+#include "vec/aggregate_functions/helpers.h"
+
+namespace doris::vectorized {
+
+namespace {
+
+template <typename T>
+struct Avg {
+ using FieldType = std::conditional_t<IsDecimalNumber<T>, Decimal128, NearestFieldType<T>>;
+ using Function = AggregateFunctionAvg<T, AggregateFunctionAvgData<FieldType>>;
+};
+
+template <typename T>
+using AggregateFuncAvg = typename Avg<T>::Function;
+
+AggregateFunctionPtr create_aggregate_function_avg(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+ assert_unary(name, argument_types);
+
+ AggregateFunctionPtr res;
+ DataTypePtr data_type = argument_types[0];
+ if (is_decimal(data_type))
+ res.reset(
+ create_with_decimal_type<AggregateFuncAvg>(*data_type, *data_type, argument_types));
+ else
+ res.reset(create_with_numeric_type<AggregateFuncAvg>(*data_type, argument_types));
+
+ if (!res) {
+ LOG(WARNING) << fmt::format("Illegal type {} of argument for aggregate function {}",
+ argument_types[0]->get_name(), name);
+ }
+ return res;
+}
+
+} // namespace
+
+void register_aggregate_function_avg(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("avg", create_aggregate_function_avg);
+}
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_avg.h b/be/src/vec/aggregate_functions/aggregate_function_avg.h
new file mode 100644
index 0000000..18584ee
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_avg.h
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionAvg.h
+// and modified by Doris
+
+#pragma once
+
+#include "common/status.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+template <typename T>
+struct AggregateFunctionAvgData {
+ T sum = 0;
+ UInt64 count = 0;
+
+ template <typename ResultT>
+ ResultT result() const {
+ if constexpr (std::is_floating_point_v<ResultT>)
+ if constexpr (std::numeric_limits<ResultT>::is_iec559)
+ return static_cast<ResultT>(sum) / count; /// allow division by zero
+
+ if (!count)
+ throw Exception("AggregateFunctionAvg with zero values", TStatusCode::VEC_LOGIC_ERROR);
+ return static_cast<ResultT>(sum) / count;
+ }
+
+ void write(BufferWritable& buf) const {
+ write_binary(sum, buf);
+ write_binary(count, buf);
+ }
+
+ void read(BufferReadable& buf) {
+ read_binary(sum, buf);
+ read_binary(count, buf);
+ }
+};
+
+/// Calculates arithmetic mean of numbers.
+template <typename T, typename Data>
+class AggregateFunctionAvg final
+ : public IAggregateFunctionDataHelper<Data, AggregateFunctionAvg<T, Data>> {
+public:
+ using ResultType = std::conditional_t<IsDecimalNumber<T>, Decimal128, Float64>;
+ using ResultDataType = std::conditional_t<IsDecimalNumber<T>, DataTypeDecimal<Decimal128>,
+ DataTypeNumber<Float64>>;
+ using ColVecType = std::conditional_t<IsDecimalNumber<T>, ColumnDecimal<T>, ColumnVector<T>>;
+ using ColVecResult = std::conditional_t<IsDecimalNumber<T>, ColumnDecimal<Decimal128>,
+ ColumnVector<Float64>>;
+
+ /// ctor for native types
+ AggregateFunctionAvg(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionAvg<T, Data>>(argument_types_,
+ {}),
+ scale(0) {}
+
+ /// ctor for Decimals
+ AggregateFunctionAvg(const IDataType& data_type, const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionAvg<T, Data>>(argument_types_,
+ {}),
+ scale(get_decimal_scale(data_type)) {}
+
+ String get_name() const override { return "avg"; }
+
+ DataTypePtr get_return_type() const override {
+ if constexpr (IsDecimalNumber<T>)
+ return std::make_shared<ResultDataType>(ResultDataType::max_precision(), scale);
+ else
+ return std::make_shared<ResultDataType>();
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ const auto& column = static_cast<const ColVecType&>(*columns[0]);
+ this->data(place).sum += column.get_data()[row_num];
+ ++this->data(place).count;
+ }
+
+ void reset(AggregateDataPtr place) const override {
+ this->data(place).sum = 0;
+ this->data(place).count = 0;
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ this->data(place).sum += this->data(rhs).sum;
+ this->data(place).count += this->data(rhs).count;
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ auto& column = static_cast<ColVecResult&>(to);
+ column.get_data().push_back(this->data(place).template result<ResultType>());
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+
+private:
+ UInt32 scale;
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_bitmap.cpp b/be/src/vec/aggregate_functions/aggregate_function_bitmap.cpp
new file mode 100644
index 0000000..d110b09
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_bitmap.cpp
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/aggregate_functions/aggregate_function_bitmap.h"
+
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+
+namespace doris::vectorized {
+
+template <bool nullable, template <bool, typename> class AggregateFunctionTemplate>
+static IAggregateFunction* createWithIntDataType(const DataTypes& argument_type) {
+ auto type = argument_type[0].get();
+ if (type->is_nullable()) {
+ type = assert_cast<const DataTypeNullable*>(type)->get_nested_type().get();
+ }
+ WhichDataType which(type);
+ if (which.idx == TypeIndex::Int8)
+ return new AggregateFunctionTemplate<nullable, ColumnVector<Int8>>(argument_type);
+ if (which.idx == TypeIndex::Int16)
+ return new AggregateFunctionTemplate<nullable, ColumnVector<Int16>>(argument_type);
+ if (which.idx == TypeIndex::Int32)
+ return new AggregateFunctionTemplate<nullable, ColumnVector<Int32>>(argument_type);
+ if (which.idx == TypeIndex::Int64)
+ return new AggregateFunctionTemplate<nullable, ColumnVector<Int64>>(argument_type);
+ return nullptr;
+}
+
+AggregateFunctionPtr create_aggregate_function_bitmap_union(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return std::make_shared<AggregateFunctionBitmapOp<AggregateFunctionBitmapUnionOp>>(
+ argument_types);
+}
+
+AggregateFunctionPtr create_aggregate_function_bitmap_intersect(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return std::make_shared<AggregateFunctionBitmapOp<AggregateFunctionBitmapIntersectOp>>(
+ argument_types);
+}
+template <bool nullable>
+AggregateFunctionPtr create_aggregate_function_bitmap_union_count(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return std::make_shared<AggregateFunctionBitmapCount<nullable, ColumnBitmap>>(argument_types);
+}
+
+template <bool nullable>
+AggregateFunctionPtr create_aggregate_function_bitmap_union_int(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return std::shared_ptr<IAggregateFunction>(
+ createWithIntDataType<nullable, AggregateFunctionBitmapCount>(argument_types));
+}
+
+void register_aggregate_function_bitmap(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("bitmap_union", create_aggregate_function_bitmap_union);
+ factory.register_function("bitmap_intersect", create_aggregate_function_bitmap_intersect);
+ factory.register_function("bitmap_union_count",
+ create_aggregate_function_bitmap_union_count<false>);
+ factory.register_function("bitmap_union_count",
+ create_aggregate_function_bitmap_union_count<true>, true);
+
+ factory.register_function("bitmap_union_int",
+ create_aggregate_function_bitmap_union_int<false>);
+ factory.register_function("bitmap_union_int", create_aggregate_function_bitmap_union_int<true>,
+ true);
+}
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/aggregate_functions/aggregate_function_bitmap.h b/be/src/vec/aggregate_functions/aggregate_function_bitmap.h
new file mode 100644
index 0000000..a2e43e5
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_bitmap.h
@@ -0,0 +1,175 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+#include <istream>
+#include <ostream>
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_complex.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/common/assert_cast.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+struct AggregateFunctionBitmapUnionOp {
+ static constexpr auto name = "bitmap_union";
+
+ template <typename T>
+ static void add(BitmapValue& res, const T& data) {
+ res.add(data);
+ }
+
+ static void add(BitmapValue& res, const BitmapValue& data) { res |= data; }
+
+ static void merge(BitmapValue& res, const BitmapValue& data) { res |= data; }
+};
+
+struct AggregateFunctionBitmapIntersectOp {
+ static constexpr auto name = "bitmap_intersect";
+ static void add(BitmapValue& res, const BitmapValue& data) { res &= data; }
+
+ static void merge(BitmapValue& res, const BitmapValue& data) { res &= data; }
+};
+
+template <typename Op>
+struct AggregateFunctionBitmapData {
+ BitmapValue value;
+
+ template <typename T>
+ void add(const T& data) {
+ Op::add(value, data);
+ }
+
+ void merge(const BitmapValue& data) { Op::merge(value, data); }
+
+ void write(BufferWritable& buf) const { DataTypeBitMap::serialize_as_stream(value, buf); }
+
+ void read(BufferReadable& buf) { DataTypeBitMap::deserialize_as_stream(value, buf); }
+
+ BitmapValue& get() { return value; }
+};
+
+template <typename Op>
+class AggregateFunctionBitmapOp final
+ : public IAggregateFunctionDataHelper<AggregateFunctionBitmapData<Op>,
+ AggregateFunctionBitmapOp<Op>> {
+public:
+ using ResultDataType = BitmapValue;
+ using ColVecType = ColumnBitmap;
+ using ColVecResult = ColumnBitmap;
+
+ String get_name() const override { return Op::name; }
+
+ AggregateFunctionBitmapOp(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<AggregateFunctionBitmapData<Op>,
+ AggregateFunctionBitmapOp<Op>>(argument_types_, {}) {}
+
+ DataTypePtr get_return_type() const override { return std::make_shared<DataTypeBitMap>(); }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ const auto& column = static_cast<const ColVecType&>(*columns[0]);
+ this->data(place).add(column.get_data()[row_num]);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ this->data(place).merge(
+ const_cast<AggregateFunctionBitmapData<Op>&>(this->data(rhs)).get());
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ auto& column = static_cast<ColVecResult&>(to);
+ column.get_data().push_back(
+ const_cast<AggregateFunctionBitmapData<Op>&>(this->data(place)).get());
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+template <bool nullable, typename ColVecType>
+class AggregateFunctionBitmapCount final
+ : public IAggregateFunctionDataHelper<
+ AggregateFunctionBitmapData<AggregateFunctionBitmapUnionOp>,
+ AggregateFunctionBitmapCount<nullable, ColVecType>> {
+public:
+ // using ColVecType = ColumnBitmap;
+ using ColVecResult = ColumnVector<Int64>;
+ using AggFunctionData = AggregateFunctionBitmapData<AggregateFunctionBitmapUnionOp>;
+
+ AggregateFunctionBitmapCount(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<
+ AggregateFunctionBitmapData<AggregateFunctionBitmapUnionOp>,
+ AggregateFunctionBitmapCount<nullable, ColVecType>>(argument_types_, {}) {}
+
+ String get_name() const override { return "count"; }
+ DataTypePtr get_return_type() const override { return std::make_shared<DataTypeInt64>(); }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ if constexpr (nullable) {
+ auto& nullable_column = assert_cast<const ColumnNullable&>(*columns[0]);
+ if (!nullable_column.is_null_at(row_num)) {
+ const auto& column =
+ static_cast<const ColVecType&>(nullable_column.get_nested_column());
+ this->data(place).add(column.get_data()[row_num]);
+ }
+ } else {
+ const auto& column = static_cast<const ColVecType&>(*columns[0]);
+ this->data(place).add(column.get_data()[row_num]);
+ }
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ this->data(place).merge(const_cast<AggFunctionData&>(this->data(rhs)).get());
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ auto& value_data = const_cast<AggFunctionData&>(this->data(place)).get();
+ auto& column = static_cast<ColVecResult&>(to);
+ column.get_data().push_back(value_data.cardinality());
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+AggregateFunctionPtr create_aggregate_function_bitmap_union(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable);
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/aggregate_functions/aggregate_function_combinator.h b/be/src/vec/aggregate_functions/aggregate_function_combinator.h
new file mode 100644
index 0000000..77d465b
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_combinator.h
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/IAggregateFunctionCombinator.h
+// and modified by Doris
+
+#pragma once
+
+#include <vec/aggregate_functions/aggregate_function.h>
+#include <vec/data_types/data_type.h>
+
+#include <memory>
+
+namespace doris::vectorized {
+
+/** Aggregate function combinator allows to take one aggregate function
+ * and transform it to another aggregate function.
+ *
+ * In SQL language they are used as suffixes for existing aggregate functions.
+ *
+ * Example: -If combinator takes an aggregate function and transforms it
+ * to aggregate function with additional argument at end (condition),
+ * that will pass values to original aggregate function when the condition is true.
+ *
+ * More examples:
+ *
+ * sum(x) - calculate sum of x
+ * sumIf(x, cond) - calculate sum of x for rows where condition is true.
+ * sumArray(arr) - calculate sum of all elements of arrays.
+ *
+ * PS. Please don't mess it with so called "combiner" - totally unrelated notion from Hadoop world.
+ * "combining" - merging the states of aggregate functions - is supported naturally in ClickHouse.
+ */
+
+class IAggregateFunctionCombinator {
+public:
+ virtual String get_name() const = 0;
+
+ virtual bool is_for_internal_usage_only() const { return false; }
+
+ /** From the arguments for combined function (ex: UInt64, UInt8 for sumIf),
+ * get the arguments for nested function (ex: UInt64 for sum).
+ * If arguments are not suitable for combined function, throw an exception.
+ */
+ virtual DataTypes transform_arguments(const DataTypes& arguments) const { return arguments; }
+
+ /** From the parameters for combined function,
+ * get the parameters for nested function.
+ * If arguments are not suitable for combined function, throw an exception.
+ */
+ virtual Array transform_parameters(const Array& parameters) const { return parameters; }
+
+ /** Create combined aggregate function (ex: sumIf)
+ * from nested function (ex: sum)
+ * and arguments for combined agggregate function (ex: UInt64, UInt8 for sumIf).
+ * It's assumed that function transform_arguments was called before this function and 'arguments' are validated.
+ */
+ virtual AggregateFunctionPtr transform_aggregate_function(
+ const AggregateFunctionPtr& nested_function, const DataTypes& arguments,
+ const Array& params, const bool result_is_nullable) const = 0;
+
+ virtual ~IAggregateFunctionCombinator() {}
+};
+
+using AggregateFunctionCombinatorPtr = std::shared_ptr<const IAggregateFunctionCombinator>;
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_count.cpp b/be/src/vec/aggregate_functions/aggregate_function_count.cpp
new file mode 100644
index 0000000..aa5d006
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_count.cpp
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionCount.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_count.h"
+
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+
+namespace doris::vectorized {
+
+AggregateFunctionPtr create_aggregate_function_count(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+ assert_arity_at_most<1>(name, argument_types);
+
+ return std::make_shared<AggregateFunctionCount>(argument_types);
+}
+
+AggregateFunctionPtr create_aggregate_function_count_not_null_unary(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_arity_at_most<1>(name, argument_types);
+
+ return std::make_shared<AggregateFunctionCountNotNullUnary>(argument_types);
+}
+
+void register_aggregate_function_count(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("count", create_aggregate_function_count);
+ factory.register_function("count", create_aggregate_function_count_not_null_unary, true);
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_count.h b/be/src/vec/aggregate_functions/aggregate_function_count.h
new file mode 100644
index 0000000..fd096dc
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_count.h
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionCount.h
+// and modified by Doris
+
+#pragma once
+
+#include <array>
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/common/assert_cast.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+struct AggregateFunctionCountData {
+ UInt64 count = 0;
+};
+
+/// Simply count number of calls.
+class AggregateFunctionCount final
+ : public IAggregateFunctionDataHelper<AggregateFunctionCountData, AggregateFunctionCount> {
+public:
+ AggregateFunctionCount(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper(argument_types_, {}) {}
+
+ String get_name() const override { return "count"; }
+
+ DataTypePtr get_return_type() const override { return std::make_shared<DataTypeInt64>(); }
+
+ void add(AggregateDataPtr __restrict place, const IColumn**, size_t, Arena*) const override {
+ ++data(place).count;
+ }
+
+ void reset(AggregateDataPtr place) const override {
+ this->data(place).count = 0;
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ data(place).count += data(rhs).count;
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ write_var_uint(data(place).count, buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ read_var_uint(data(place).count, buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ assert_cast<ColumnInt64&>(to).get_data().push_back(data(place).count);
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+/// Simply count number of not-NULL values.
+class AggregateFunctionCountNotNullUnary final
+ : public IAggregateFunctionDataHelper<AggregateFunctionCountData,
+ AggregateFunctionCountNotNullUnary> {
+public:
+ AggregateFunctionCountNotNullUnary(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper(argument_types_, {}) {}
+
+ String get_name() const override { return "count"; }
+
+ DataTypePtr get_return_type() const override { return std::make_shared<DataTypeInt64>(); }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ data(place).count += !assert_cast<const ColumnNullable&>(*columns[0]).is_null_at(row_num);
+ }
+
+ void reset(AggregateDataPtr place) const override {
+ data(place).count = 0;
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ data(place).count += data(rhs).count;
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ write_var_uint(data(place).count, buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ read_var_uint(data(place).count, buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ if (to.is_nullable()) {
+ auto& null_column = assert_cast<ColumnNullable &>(to);
+ null_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64 &>(null_column.get_nested_column()).get_data().push_back(data(place).count);
+ } else {
+ assert_cast<ColumnInt64 &>(to).get_data().push_back(data(place).count);
+ }
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_distinct.cpp b/be/src/vec/aggregate_functions/aggregate_function_distinct.cpp
new file mode 100644
index 0000000..af64a3a
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_distinct.cpp
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionDistinct.cpp
+// and modified by Doris
+
+#include <algorithm>
+#include <boost/algorithm/string.hpp>
+
+#include "vec/aggregate_functions/aggregate_function_combinator.h"
+#include "vec/aggregate_functions/aggregate_function_distinct.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/helpers.h"
+#include "vec/common/typeid_cast.h"
+#include "vec/data_types/data_type_nullable.h"
+
+namespace doris::vectorized {
+
+class AggregateFunctionCombinatorDistinct final : public IAggregateFunctionCombinator {
+public:
+ String get_name() const override { return "Distinct"; }
+
+ DataTypes transform_arguments(const DataTypes& arguments) const override {
+ if (arguments.empty()) {
+ LOG(FATAL)
+ << "Incorrect number of arguments for aggregate function with Distinct suffix";
+ }
+ return arguments;
+ }
+
+ AggregateFunctionPtr transform_aggregate_function(const AggregateFunctionPtr& nested_function,
+ const DataTypes& arguments,
+ const Array& params,
+ const bool result_is_nullable) const override {
+ DCHECK(nested_function != nullptr);
+ if (nested_function == nullptr) return nullptr;
+
+ AggregateFunctionPtr res;
+ if (arguments.size() == 1) {
+ res.reset(create_with_numeric_type<AggregateFunctionDistinct,
+ AggregateFunctionDistinctSingleNumericData>(
+ *arguments[0], nested_function, arguments));
+
+ if (res) return res;
+
+ if (arguments[0]->is_value_unambiguously_represented_in_contiguous_memory_region())
+ return std::make_shared<AggregateFunctionDistinct<
+ AggregateFunctionDistinctSingleGenericData<true>>>(nested_function,
+ arguments);
+ else
+ return std::make_shared<AggregateFunctionDistinct<
+ AggregateFunctionDistinctSingleGenericData<false>>>(nested_function,
+ arguments);
+ }
+
+ return std::make_shared<
+ AggregateFunctionDistinct<AggregateFunctionDistinctMultipleGenericData>>(
+ nested_function, arguments);
+ }
+};
+
+const std::string DISTINCT_FUNCTION_PREFIX = "multi_distinct_";
+
+void register_aggregate_function_combinator_distinct(AggregateFunctionSimpleFactory& factory) {
+ AggregateFunctionCreator creator = [&](const std::string& name, const DataTypes& types,
+ const Array& params, const bool result_is_nullable) {
+ // 1. we should get not nullable types;
+ DataTypes nested_types(types.size());
+ std::transform(types.begin(), types.end(), nested_types.begin(),
+ [](const auto& e) { return remove_nullable(e); });
+ auto function_combinator = std::make_shared<AggregateFunctionCombinatorDistinct>();
+ auto transform_arguments = function_combinator->transform_arguments(nested_types);
+ if (!boost::algorithm::starts_with(name, DISTINCT_FUNCTION_PREFIX)) {
+ return AggregateFunctionPtr();
+ }
+ auto nested_function_name = name.substr(DISTINCT_FUNCTION_PREFIX.size());
+ auto nested_function = factory.get(nested_function_name, transform_arguments, params);
+ return function_combinator->transform_aggregate_function(nested_function, types, params, result_is_nullable);
+ };
+ factory.register_distinct_function_combinator(creator, DISTINCT_FUNCTION_PREFIX);
+}
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_distinct.h b/be/src/vec/aggregate_functions/aggregate_function_distinct.h
new file mode 100644
index 0000000..6502b48
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_distinct.h
@@ -0,0 +1,224 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionDistinct.h
+// and modified by Doris
+
+#pragma once
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/aggregate_functions/key_holder_helpers.h"
+#include "vec/common/aggregation_common.h"
+#include "vec/common/assert_cast.h"
+#include "vec/common/field_visitors.h"
+#include "vec/common/hash_table/hash_set.h"
+#include "vec/common/hash_table/hash_table.h"
+#include "vec/common/sip_hash.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+template <typename T>
+struct AggregateFunctionDistinctSingleNumericData {
+ /// When creating, the hash table must be small.
+ using Set = HashSetWithStackMemory<T, DefaultHash<T>, 4>;
+ using Self = AggregateFunctionDistinctSingleNumericData<T>;
+ Set set;
+
+ void add(const IColumn** columns, size_t /* columns_num */, size_t row_num, Arena*) {
+ const auto& vec = assert_cast<const ColumnVector<T>&>(*columns[0]).get_data();
+ set.insert(vec[row_num]);
+ }
+
+ void merge(const Self& rhs, Arena*) { set.merge(rhs.set); }
+
+ void serialize(BufferWritable& buf) const { set.write(buf); }
+
+ void deserialize(BufferReadable& buf, Arena*) { set.read(buf); }
+
+ MutableColumns get_arguments(const DataTypes& argument_types) const {
+ MutableColumns argument_columns;
+ argument_columns.emplace_back(argument_types[0]->create_column());
+ for (const auto& elem : set) argument_columns[0]->insert(elem.get_value());
+
+ return argument_columns;
+ }
+};
+
+struct AggregateFunctionDistinctGenericData {
+ /// When creating, the hash table must be small.
+ using Set = HashSetWithSavedHashWithStackMemory<StringRef, StringRefHash, 4>;
+ using Self = AggregateFunctionDistinctGenericData;
+ Set set;
+
+ void merge(const Self& rhs, Arena* arena) {
+ Set::LookupResult it;
+ bool inserted;
+ for (const auto& elem : rhs.set)
+ set.emplace(ArenaKeyHolder{elem.get_value(), *arena}, it, inserted);
+ }
+
+ void serialize(BufferWritable& buf) const {
+ write_var_uint(set.size(), buf);
+ for (const auto& elem : set)
+ write_string_binary(elem.get_value(), buf);
+ }
+
+ void deserialize(BufferReadable& buf, Arena* arena) {
+ size_t size;
+ read_var_uint(size, buf);
+
+ StringRef ref;
+ for (size_t i = 0; i < size; ++i) {
+ read_string_binary(ref, buf);
+ set.insert(ref);
+ }
+ }
+};
+
+template <bool is_plain_column>
+struct AggregateFunctionDistinctSingleGenericData : public AggregateFunctionDistinctGenericData {
+ void add(const IColumn** columns, size_t /* columns_num */, size_t row_num, Arena* arena) {
+ Set::LookupResult it;
+ bool inserted;
+ auto key_holder = get_key_holder<is_plain_column>(*columns[0], row_num, *arena);
+ set.emplace(key_holder, it, inserted);
+ }
+
+ MutableColumns get_arguments(const DataTypes& argument_types) const {
+ MutableColumns argument_columns;
+ argument_columns.emplace_back(argument_types[0]->create_column());
+ for (const auto& elem : set)
+ deserialize_and_insert<is_plain_column>(elem.get_value(), *argument_columns[0]);
+
+ return argument_columns;
+ }
+};
+
+struct AggregateFunctionDistinctMultipleGenericData : public AggregateFunctionDistinctGenericData {
+ void add(const IColumn** columns, size_t columns_num, size_t row_num, Arena* arena) {
+ const char* begin = nullptr;
+ StringRef value(begin, 0);
+ for (size_t i = 0; i < columns_num; ++i) {
+ auto cur_ref = columns[i]->serialize_value_into_arena(row_num, *arena, begin);
+ value.data = cur_ref.data - value.size;
+ value.size += cur_ref.size;
+ }
+
+ Set::LookupResult it;
+ bool inserted;
+ auto key_holder = SerializedKeyHolder{value, *arena};
+ set.emplace(key_holder, it, inserted);
+ }
+
+ MutableColumns get_arguments(const DataTypes& argument_types) const {
+ MutableColumns argument_columns(argument_types.size());
+ for (size_t i = 0; i < argument_types.size(); ++i)
+ argument_columns[i] = argument_types[i]->create_column();
+
+ for (const auto& elem : set) {
+ const char* begin = elem.get_value().data;
+ for (auto& column : argument_columns)
+ begin = column->deserialize_and_insert_from_arena(begin);
+ }
+
+ return argument_columns;
+ }
+};
+
+/** Adaptor for aggregate functions.
+ * Adding -Distinct suffix to aggregate function
+**/
+template <typename Data>
+class AggregateFunctionDistinct
+ : public IAggregateFunctionDataHelper<Data, AggregateFunctionDistinct<Data>> {
+private:
+ static constexpr auto prefix_size = sizeof(Data);
+ AggregateFunctionPtr nested_func;
+ size_t arguments_num;
+
+ AggregateDataPtr get_nested_place(AggregateDataPtr __restrict place) const noexcept {
+ return place + prefix_size;
+ }
+
+ ConstAggregateDataPtr get_nested_place(ConstAggregateDataPtr __restrict place) const noexcept {
+ return place + prefix_size;
+ }
+
+public:
+ AggregateFunctionDistinct(AggregateFunctionPtr nested_func_, const DataTypes& arguments)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionDistinct>(
+ arguments, nested_func_->get_parameters()),
+ nested_func(nested_func_),
+ arguments_num(arguments.size()) {}
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena* arena) const override {
+ this->data(place).add(columns, arguments_num, row_num, arena);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena* arena) const override {
+ this->data(place).merge(this->data(rhs), arena);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).serialize(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena* arena) const override {
+ this->data(place).deserialize(buf, arena);
+ }
+
+ // void insert_result_into(AggregateDataPtr place, IColumn & to, Arena * arena) const override
+ void insert_result_into(ConstAggregateDataPtr targetplace, IColumn& to) const override {
+ auto place = const_cast<AggregateDataPtr>(targetplace);
+ auto arguments = this->data(place).get_arguments(this->argument_types);
+ ColumnRawPtrs arguments_raw(arguments.size());
+ for (size_t i = 0; i < arguments.size(); ++i) arguments_raw[i] = arguments[i].get();
+
+ assert(!arguments.empty());
+ // nested_func->add_batch_single_place(arguments[0]->size(), get_nested_place(place), arguments_raw.data(), arena);
+ // nested_func->insert_result_into(get_nested_place(place), to, arena);
+
+ nested_func->add_batch_single_place(arguments[0]->size(), get_nested_place(place),
+ arguments_raw.data(), nullptr);
+ nested_func->insert_result_into(get_nested_place(place), to);
+ }
+
+ size_t size_of_data() const override { return prefix_size + nested_func->size_of_data(); }
+
+ void create(AggregateDataPtr __restrict place) const override {
+ new (place) Data;
+ nested_func->create(get_nested_place(place));
+ }
+
+ void destroy(AggregateDataPtr __restrict place) const noexcept override {
+ this->data(place).~Data();
+ nested_func->destroy(get_nested_place(place));
+ }
+
+ String get_name() const override { return nested_func->get_name() + "Distinct"; }
+
+ DataTypePtr get_return_type() const override { return nested_func->get_return_type(); }
+
+ bool allocates_memory_in_arena() const override { return true; }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.cpp b/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.cpp
new file mode 100644
index 0000000..3b2aba0
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.cpp
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/aggregate_functions/aggregate_function_hll_union_agg.h"
+
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+
+namespace doris::vectorized {
+
+AggregateFunctionPtr create_aggregate_function_HLL_union_agg(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+ assert_arity_at_most<1>(name, argument_types);
+
+ return std::make_shared<AggregateFunctionHLLUnionAgg>(argument_types);
+}
+
+AggregateFunctionPtr create_aggregate_function_HLL_union(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+ assert_arity_at_most<1>(name, argument_types);
+
+ return std::make_shared<AggregateFunctionHLLUnion>(argument_types);
+}
+
+void register_aggregate_function_HLL_union_agg(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("hll_union_agg", create_aggregate_function_HLL_union_agg);
+ factory.register_function("hll_union", create_aggregate_function_HLL_union);
+ factory.register_function("hll_raw_agg", create_aggregate_function_HLL_union);
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h b/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
new file mode 100644
index 0000000..f71a1f5
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <istream>
+#include <ostream>
+#include <type_traits>
+
+#include "exprs/hll_function.h"
+#include "olap/hll.h"
+#include "util/slice.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+struct AggregateFunctionHLLData {
+ HyperLogLog dst_hll {};
+
+ void add(const StringRef& src) { dst_hll.merge(HyperLogLog(Slice(src.data, src.size))); }
+
+ void merge(const AggregateFunctionHLLData& rhs) { dst_hll.merge(rhs.dst_hll); }
+
+ void write(BufferWritable& buf) const {
+ std::string result(dst_hll.max_serialized_size(), '0');
+ int size = dst_hll.serialize((uint8_t*)result.c_str());
+ result.resize(size);
+ write_binary(result, buf);
+ }
+
+ void read(BufferReadable& buf) {
+ StringRef ref;
+ read_binary(ref, buf);
+ dst_hll.deserialize(Slice(ref.data, ref.size));
+ }
+
+ Int64 get_cardinality() const { return dst_hll.estimate_cardinality(); }
+
+ std::string get() const {
+ std::string result(dst_hll.max_serialized_size(), '0');
+ int size = dst_hll.serialize((uint8_t*)result.c_str());
+ result.resize(size);
+
+ return result;
+ }
+};
+
+class AggregateFunctionHLLUnionAgg
+ : public IAggregateFunctionDataHelper<AggregateFunctionHLLData,
+ AggregateFunctionHLLUnionAgg> {
+public:
+ virtual String get_name() const override { return "hll_union_agg"; }
+
+ AggregateFunctionHLLUnionAgg(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper(argument_types_, {}) {}
+
+ AggregateFunctionHLLUnionAgg(const IDataType& data_type, const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper(argument_types_, {}) {}
+
+ virtual DataTypePtr get_return_type() const override {
+ return std::make_shared<DataTypeInt64>();
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ const auto& column = static_cast<const ColumnString&>(*columns[0]);
+ this->data(place).add(column.get_data_at(row_num));
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ this->data(place).merge(this->data(rhs));
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ virtual void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ auto& column = static_cast<ColumnVector<Int64>&>(to);
+ column.get_data().push_back(this->data(place).get_cardinality());
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+class AggregateFunctionHLLUnion final : public AggregateFunctionHLLUnionAgg {
+public:
+ String get_name() const override { return "hll_union"; }
+
+ AggregateFunctionHLLUnion(const DataTypes& argument_types_)
+ : AggregateFunctionHLLUnionAgg {argument_types_} {}
+
+ AggregateFunctionHLLUnion(const IDataType& data_type, const DataTypes& argument_types_)
+ : AggregateFunctionHLLUnionAgg(data_type, argument_types_) {}
+
+ DataTypePtr get_return_type() const override { return std::make_shared<DataTypeString>(); }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ auto& column = static_cast<ColumnString&>(to);
+ auto result = this->data(place).get();
+ column.insert_data(result.c_str(), result.length());
+ }
+};
+
+AggregateFunctionPtr create_aggregate_function_HLL_union(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable);
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_min_max.cpp b/be/src/vec/aggregate_functions/aggregate_function_min_max.cpp
new file mode 100644
index 0000000..813674a
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_min_max.cpp
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionMinMaxAny.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_min_max.h"
+
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+#include "vec/aggregate_functions/helpers.h"
+
+namespace doris::vectorized {
+
+/// min, max
+template <template <typename, bool> class AggregateFunctionTemplate, template <typename> class Data>
+static IAggregateFunction* create_aggregate_function_single_value(const String& name,
+ const DataTypes& argument_types,
+ const Array& parameters) {
+ assert_no_parameters(name, parameters);
+ assert_unary(name, argument_types);
+
+ const DataTypePtr& argument_type = argument_types[0];
+
+ WhichDataType which(argument_type);
+#define DISPATCH(TYPE) \
+ if (which.idx == TypeIndex::TYPE) \
+ return new AggregateFunctionTemplate<Data<SingleValueDataFixed<TYPE>>, false>( \
+ argument_type);
+ FOR_NUMERIC_TYPES(DISPATCH)
+#undef DISPATCH
+ if (which.idx == TypeIndex::String) {
+ return new AggregateFunctionTemplate<Data<SingleValueDataString>, false>(argument_type);
+ }
+ if (which.idx == TypeIndex::DateTime || which.idx == TypeIndex::Date) {
+ return new AggregateFunctionTemplate<Data<SingleValueDataFixed<Int64>>, false>(
+ argument_type);
+ }
+ if (which.idx == TypeIndex::Decimal128) {
+ return new AggregateFunctionTemplate<Data<SingleValueDataFixed<DecimalV2Value>>, false>(
+ argument_type);
+ }
+ return nullptr;
+}
+
+AggregateFunctionPtr create_aggregate_function_max(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_aggregate_function_single_value<AggregateFunctionsSingleValue,
+ AggregateFunctionMaxData>(name, argument_types,
+ parameters));
+}
+
+AggregateFunctionPtr create_aggregate_function_min(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_aggregate_function_single_value<AggregateFunctionsSingleValue,
+ AggregateFunctionMinData>(name, argument_types,
+ parameters));
+}
+
+void register_aggregate_function_minmax(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("max", create_aggregate_function_max);
+ factory.register_function("min", create_aggregate_function_min);
+}
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/aggregate_functions/aggregate_function_min_max.h b/be/src/vec/aggregate_functions/aggregate_function_min_max.h
new file mode 100644
index 0000000..17d6823
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_min_max.h
@@ -0,0 +1,535 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionMinMaxAny.h
+// and modified by Doris
+
+#pragma once
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_vector.h"
+#include "vec/common/assert_cast.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+/// For numeric values.
+template <typename T>
+struct SingleValueDataFixed {
+private:
+ using Self = SingleValueDataFixed;
+
+ bool has_value = false; /// We need to remember if at least one value has been passed. This is necessary for AggregateFunctionIf.
+ T value;
+
+public:
+ bool has() const { return has_value; }
+
+ void insert_result_into(IColumn& to) const {
+ if (has())
+ assert_cast<ColumnVector<T>&>(to).get_data().push_back(value);
+ else
+ assert_cast<ColumnVector<T>&>(to).insert_default();
+ }
+
+ void reset() {
+ if (has()) {
+ has_value = false;
+ }
+ }
+
+ void write(BufferWritable& buf) const {
+ write_binary(has(), buf);
+ if (has()) write_binary(value, buf);
+ }
+
+ void read(BufferReadable& buf) {
+ read_binary(has_value, buf);
+ if (has()) read_binary(value, buf);
+ }
+
+ void change(const IColumn& column, size_t row_num, Arena*) {
+ has_value = true;
+ value = assert_cast<const ColumnVector<T>&>(column).get_data()[row_num];
+ }
+
+ /// Assuming to.has()
+ void change(const Self& to, Arena*) {
+ has_value = true;
+ value = to.value;
+ }
+
+ bool change_first_time(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has()) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_first_time(const Self& to, Arena* arena) {
+ if (!has() && to.has()) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_every_time(const IColumn& column, size_t row_num, Arena* arena) {
+ change(column, row_num, arena);
+ return true;
+ }
+
+ bool change_every_time(const Self& to, Arena* arena) {
+ if (to.has()) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_less(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has() || assert_cast<const ColumnVector<T>&>(column).get_data()[row_num] < value) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_less(const Self& to, Arena* arena) {
+ if (to.has() && (!has() || to.value < value)) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_greater(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has() || assert_cast<const ColumnVector<T>&>(column).get_data()[row_num] > value) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_greater(const Self& to, Arena* arena) {
+ if (to.has() && (!has() || to.value > value)) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool is_equal_to(const Self& to) const { return has() && to.value == value; }
+
+ bool is_equal_to(const IColumn& column, size_t row_num) const {
+ return has() && assert_cast<const ColumnVector<T>&>(column).get_data()[row_num] == value;
+ }
+};
+
+/// For numeric values.
+template <>
+struct SingleValueDataFixed<DecimalV2Value> {
+private:
+ using Self = SingleValueDataFixed;
+
+ bool has_value =
+ false; /// We need to remember if at least one value has been passed. This is necessary for AggregateFunctionIf.
+ int128_t value;
+
+public:
+ bool has() const { return has_value; }
+
+ void insert_result_into(IColumn& to) const {
+ if (has()) {
+ DecimalV2Value decimal(value);
+ assert_cast<ColumnDecimal<Decimal128>&>(to).insert_data((const char*)&decimal, 0);
+ } else
+ assert_cast<ColumnDecimal<Decimal128>&>(to).insert_default();
+ }
+
+ void reset() {
+ if (has()) {
+ has_value = false;
+ }
+ }
+
+ void write(BufferWritable& buf) const {
+ write_binary(has(), buf);
+ if (has()) write_binary(value, buf);
+ }
+
+ void read(BufferReadable& buf) {
+ read_binary(has_value, buf);
+ if (has()) read_binary(value, buf);
+ }
+
+ void change(const IColumn& column, size_t row_num, Arena*) {
+ has_value = true;
+ value = assert_cast<const ColumnDecimal<Decimal128>&>(column).get_data()[row_num];
+ }
+
+ /// Assuming to.has()
+ void change(const Self& to, Arena*) {
+ has_value = true;
+ value = to.value;
+ }
+
+ bool change_first_time(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has()) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_first_time(const Self& to, Arena* arena) {
+ if (!has() && to.has()) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_every_time(const IColumn& column, size_t row_num, Arena* arena) {
+ change(column, row_num, arena);
+ return true;
+ }
+
+ bool change_every_time(const Self& to, Arena* arena) {
+ if (to.has()) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_less(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has() ||
+ assert_cast<const ColumnDecimal<Decimal128>&>(column).get_data()[row_num] < value) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_less(const Self& to, Arena* arena) {
+ if (to.has() && (!has() || to.value < value)) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_greater(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has() ||
+ assert_cast<const ColumnDecimal<Decimal128>&>(column).get_data()[row_num] > value) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_greater(const Self& to, Arena* arena) {
+ if (to.has() && (!has() || to.value > value)) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool is_equal_to(const Self& to) const { return has() && to.value == value; }
+
+ bool is_equal_to(const IColumn& column, size_t row_num) const {
+ return has() &&
+ assert_cast<const ColumnDecimal<Decimal128>&>(column).get_data()[row_num] == value;
+ }
+};
+
+/** For strings. Short strings are stored in the object itself, and long strings are allocated separately.
+ * NOTE It could also be suitable for arrays of numbers.
+ */
+struct SingleValueDataString {
+private:
+ using Self = SingleValueDataString;
+
+ Int32 size = -1; /// -1 indicates that there is no value.
+ Int32 capacity = 0; /// power of two or zero
+ char* large_data = nullptr;
+
+public:
+ static constexpr Int32 AUTOMATIC_STORAGE_SIZE = 64;
+ static constexpr Int32 MAX_SMALL_STRING_SIZE =
+ AUTOMATIC_STORAGE_SIZE - sizeof(size) - sizeof(capacity) - sizeof(large_data);
+
+private:
+ char small_data[MAX_SMALL_STRING_SIZE]; /// Including the terminating zero.
+
+public:
+ ~SingleValueDataString() { delete large_data; }
+
+ bool has() const { return size >= 0; }
+
+ const char* get_data() const { return size <= MAX_SMALL_STRING_SIZE ? small_data : large_data; }
+
+ void insert_result_into(IColumn& to) const {
+ if (has())
+ assert_cast<ColumnString&>(to).insert_data(get_data(), size);
+ else
+ assert_cast<ColumnString&>(to).insert_default();
+ }
+
+ void reset() {
+ if (size != -1) {
+ size = -1;
+ capacity = 0;
+ delete large_data;
+ large_data = nullptr;
+ }
+ }
+
+ void write(BufferWritable& buf) const {
+ write_binary(size, buf);
+ if (has()) buf.write(get_data(), size);
+ }
+
+ void read(BufferReadable& buf) {
+ Int32 rhs_size;
+ read_binary(rhs_size, buf);
+
+ if (rhs_size >= 0) {
+ if (rhs_size <= MAX_SMALL_STRING_SIZE) {
+ /// Don't free large_data here.
+
+ size = rhs_size;
+
+ if (size > 0) buf.read(small_data, size);
+ } else {
+ if (capacity < rhs_size) {
+ capacity = static_cast<UInt32>(round_up_to_power_of_two_or_zero(rhs_size));
+ delete large_data;
+ large_data = new char[capacity];
+ }
+
+ size = rhs_size;
+ buf.read(large_data, size);
+ }
+ } else {
+ /// Don't free large_data here.
+ size = rhs_size;
+ }
+ }
+
+ StringRef get_string_ref() const { return StringRef(get_data(), size); }
+
+ /// Assuming to.has()
+ void change_impl(StringRef value, Arena* arena) {
+ Int32 value_size = value.size;
+
+ if (value_size <= MAX_SMALL_STRING_SIZE) {
+ /// Don't free large_data here.
+ size = value_size;
+
+ if (size > 0) {
+ memcpy(small_data, value.data, size);
+ }
+ } else {
+ if (capacity < value_size) {
+ /// Don't free large_data here.
+ capacity = round_up_to_power_of_two_or_zero(value_size);
+ delete large_data;
+ large_data = new char[capacity];
+ }
+
+ size = value_size;
+ memcpy(large_data, value.data, size);
+ }
+ }
+
+ void change(const IColumn& column, size_t row_num, Arena* arena) {
+ change_impl(assert_cast<const ColumnString&>(column).get_data_at(row_num), arena);
+ }
+
+ void change(const Self& to, Arena* arena) { change_impl(to.get_string_ref(), arena); }
+
+ bool change_if_less(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has() ||
+ assert_cast<const ColumnString&>(column).get_data_at(row_num) < get_string_ref()) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_greater(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has() ||
+ assert_cast<const ColumnString&>(column).get_data_at(row_num) > get_string_ref()) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_first_time(const IColumn& column, size_t row_num, Arena* arena) {
+ if (!has()) {
+ change(column, row_num, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_first_time(const Self& to, Arena* arena) {
+ if (!has() && to.has()) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_every_time(const IColumn& column, size_t row_num, Arena* arena) {
+ change(column, row_num, arena);
+ return true;
+ }
+
+ bool change_every_time(const Self& to, Arena* arena) {
+ if (to.has()) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_less(const Self& to, Arena* arena) {
+ if (to.has() && (!has() || to.get_string_ref() < get_string_ref())) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool change_if_greater(const Self& to, Arena* arena) {
+ if (to.has() && (!has() || to.get_string_ref() > get_string_ref())) {
+ change(to, arena);
+ return true;
+ } else
+ return false;
+ }
+
+ bool is_equal_to(const Self& to) const {
+ return has() && to.get_string_ref() == get_string_ref();
+ }
+
+ bool is_equal_to(const IColumn& column, size_t row_num) const { return false; }
+};
+
+template <typename Data>
+struct AggregateFunctionMaxData : Data {
+ using Self = AggregateFunctionMaxData;
+
+ bool change_if_better(const IColumn& column, size_t row_num, Arena* arena) {
+ return this->change_if_greater(column, row_num, arena);
+ }
+ bool change_if_better(const Self& to, Arena* arena) {
+ return this->change_if_greater(to, arena);
+ }
+
+ static const char* name() { return "max"; }
+};
+
+template <typename Data>
+struct AggregateFunctionMinData : Data {
+ using Self = AggregateFunctionMinData;
+
+ bool change_if_better(const IColumn& column, size_t row_num, Arena* arena) {
+ return this->change_if_less(column, row_num, arena);
+ }
+ bool change_if_better(const Self& to, Arena* arena) { return this->change_if_less(to, arena); }
+
+ static const char* name() { return "min"; }
+};
+
+template <typename Data, bool AllocatesMemoryInArena>
+class AggregateFunctionsSingleValue final
+ : public IAggregateFunctionDataHelper<
+ Data, AggregateFunctionsSingleValue<Data, AllocatesMemoryInArena>> {
+private:
+ DataTypePtr& type;
+
+public:
+ AggregateFunctionsSingleValue(const DataTypePtr& type_)
+ : IAggregateFunctionDataHelper<
+ Data, AggregateFunctionsSingleValue<Data, AllocatesMemoryInArena>>({type_},
+ {}),
+ type(this->argument_types[0]) {
+ if (StringRef(Data::name()) == StringRef("min") ||
+ StringRef(Data::name()) == StringRef("max")) {
+ if (!type->is_comparable()) {
+ LOG(FATAL) << fmt::format(
+ "Illegal type {} of argument of aggregate function {} because the values "
+ "of that data type are not comparable",
+ type->get_name(), get_name());
+ }
+ }
+ }
+
+ String get_name() const override { return Data::name(); }
+
+ DataTypePtr get_return_type() const override { return type; }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena* arena) const override {
+ this->data(place).change_if_better(*columns[0], row_num, arena);
+ }
+
+ void reset(AggregateDataPtr place) const override {
+ this->data(place).reset();
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena* arena) const override {
+ this->data(place).change_if_better(this->data(rhs), arena);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ bool allocates_memory_in_arena() const override { return AllocatesMemoryInArena; }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ this->data(place).insert_result_into(to);
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+AggregateFunctionPtr create_aggregate_function_max(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable);
+
+AggregateFunctionPtr create_aggregate_function_min(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable);
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/aggregate_functions/aggregate_function_nothing.h b/be/src/vec/aggregate_functions/aggregate_function_nothing.h
new file mode 100644
index 0000000..4c7b193
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_nothing.h
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionNothing.h
+// and modified by Doris
+
+#pragma once
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column.h"
+#include "vec/data_types/data_type_nothing.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+/** Aggregate function that takes arbitrary number of arbitrary arguments and does nothing.
+ */
+class AggregateFunctionNothing final : public IAggregateFunctionHelper<AggregateFunctionNothing> {
+public:
+ AggregateFunctionNothing(const DataTypes& arguments, const Array& params)
+ : IAggregateFunctionHelper<AggregateFunctionNothing>(arguments, params) {}
+
+ String get_name() const override { return "nothing"; }
+
+ DataTypePtr get_return_type() const override {
+ return std::make_shared<DataTypeNullable>(std::make_shared<DataTypeNothing>());
+ }
+
+ void create(AggregateDataPtr) const override {}
+
+ void destroy(AggregateDataPtr) const noexcept override {}
+
+ bool has_trivial_destructor() const override { return true; }
+
+ size_t size_of_data() const override { return 0; }
+
+ size_t align_of_data() const override { return 1; }
+
+ void add(AggregateDataPtr, const IColumn**, size_t, Arena*) const override {}
+
+ void reset(AggregateDataPtr place) const override {}
+
+ void merge(AggregateDataPtr, ConstAggregateDataPtr, Arena*) const override {}
+
+ void serialize(ConstAggregateDataPtr, BufferWritable& buf) const override {}
+
+ void deserialize(AggregateDataPtr, BufferReadable& buf, Arena*) const override {}
+
+ void insert_result_into(ConstAggregateDataPtr, IColumn& to) const override {
+ to.insert_default();
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_null.cpp b/be/src/vec/aggregate_functions/aggregate_function_null.cpp
new file mode 100644
index 0000000..e28287f
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_null.cpp
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionNull.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_null.h"
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/aggregate_functions/aggregate_function_combinator.h"
+#include "vec/aggregate_functions/aggregate_function_count.h"
+#include "vec/aggregate_functions/aggregate_function_nothing.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/data_types/data_type_nullable.h"
+
+namespace doris::vectorized {
+
+class AggregateFunctionCombinatorNull final : public IAggregateFunctionCombinator {
+public:
+ String get_name() const override { return "Null"; }
+
+ bool is_for_internal_usage_only() const override { return true; }
+
+ DataTypes transform_arguments(const DataTypes& arguments) const override {
+ size_t size = arguments.size();
+ DataTypes res(size);
+ for (size_t i = 0; i < size; ++i) res[i] = remove_nullable(arguments[i]);
+ return res;
+ }
+
+ AggregateFunctionPtr transform_aggregate_function(const AggregateFunctionPtr& nested_function,
+ const DataTypes& arguments,
+ const Array& params,
+ const bool result_is_nullable) const override {
+ if (nested_function == nullptr) return nullptr;
+
+ bool has_null_types = false;
+ for (const auto& arg_type : arguments) {
+ if (arg_type->only_null()) {
+ has_null_types = true;
+ break;
+ }
+ }
+
+ if (has_null_types) return std::make_shared<AggregateFunctionNothing>(arguments, params);
+
+ if (arguments.size() == 1) {
+ if (result_is_nullable)
+ return std::make_shared<AggregateFunctionNullUnary<true>>(nested_function,
+ arguments, params);
+ else
+ return std::make_shared<AggregateFunctionNullUnary<false>>(nested_function,
+ arguments, params);
+ } else {
+ if (result_is_nullable)
+ return std::make_shared<AggregateFunctionNullVariadic<true>>(nested_function,
+ arguments, params);
+ else
+ return std::make_shared<AggregateFunctionNullVariadic<false>>(nested_function,
+ arguments, params);
+ }
+ }
+};
+
+void register_aggregate_function_combinator_null(AggregateFunctionSimpleFactory& factory) {
+ // factory.registerCombinator(std::make_shared<AggregateFunctionCombinatorNull>());
+ AggregateFunctionCreator creator = [&](const std::string& name, const DataTypes& types,
+ const Array& params, const bool result_is_nullable) {
+ auto function_combinator = std::make_shared<AggregateFunctionCombinatorNull>();
+ auto transform_arguments = function_combinator->transform_arguments(types);
+ auto nested_function = factory.get(name, transform_arguments, params);
+ return function_combinator->transform_aggregate_function(nested_function, types, params, result_is_nullable);
+ };
+ factory.register_nullable_function_combinator(creator);
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_null.h b/be/src/vec/aggregate_functions/aggregate_function_null.h
new file mode 100644
index 0000000..4c61e2b
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_null.h
@@ -0,0 +1,291 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionNull.h
+// and modified by Doris
+
+#pragma once
+
+#include <array>
+
+#include "common/logging.h"
+#include "common/status.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/common/assert_cast.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+/// This class implements a wrapper around an aggregate function. Despite its name,
+/// this is an adapter. It is used to handle aggregate functions that are called with
+/// at least one nullable argument. It implements the logic according to which any
+/// row that contains at least one NULL is skipped.
+
+/// If all rows had NULL, the behaviour is determined by "result_is_nullable" template parameter.
+/// true - return NULL; false - return value from empty aggregation state of nested function.
+
+template <bool result_is_nullable, typename Derived>
+class AggregateFunctionNullBase : public IAggregateFunctionHelper<Derived> {
+protected:
+ AggregateFunctionPtr nested_function;
+ size_t prefix_size;
+
+ /** In addition to data for nested aggregate function, we keep a flag
+ * indicating - was there at least one non-NULL value accumulated.
+ * In case of no not-NULL values, the function will return NULL.
+ *
+ * We use prefix_size bytes for flag to satisfy the alignment requirement of nested state.
+ */
+
+ AggregateDataPtr nested_place(AggregateDataPtr __restrict place) const noexcept {
+ return place + prefix_size;
+ }
+
+ ConstAggregateDataPtr nested_place(ConstAggregateDataPtr __restrict place) const noexcept {
+ return place + prefix_size;
+ }
+
+ static void init_flag(AggregateDataPtr __restrict place) noexcept {
+ if constexpr (result_is_nullable)
+ place[0] = 0;
+ }
+
+ static void set_flag(AggregateDataPtr __restrict place) noexcept {
+ if constexpr (result_is_nullable)
+ place[0] = 1;
+ }
+
+ static bool get_flag(ConstAggregateDataPtr __restrict place) noexcept {
+ return result_is_nullable ? place[0] : 1;
+ }
+
+public:
+ AggregateFunctionNullBase(AggregateFunctionPtr nested_function_, const DataTypes& arguments,
+ const Array& params)
+ : IAggregateFunctionHelper<Derived>(arguments, params),
+ nested_function {nested_function_} {
+ if (result_is_nullable)
+ prefix_size = nested_function->align_of_data();
+ else
+ prefix_size = 0;
+ }
+
+ String get_name() const override {
+ /// This is just a wrapper. The function for Nullable arguments is named the same as the nested function itself.
+ return nested_function->get_name();
+ }
+
+ DataTypePtr get_return_type() const override {
+ return result_is_nullable ? make_nullable(nested_function->get_return_type())
+ : nested_function->get_return_type();
+ }
+
+ void create(AggregateDataPtr __restrict place) const override {
+ init_flag(place);
+ nested_function->create(nested_place(place));
+ }
+
+ void destroy(AggregateDataPtr __restrict place) const noexcept override {
+ nested_function->destroy(nested_place(place));
+ }
+ void reset(AggregateDataPtr place) const override {
+ init_flag(place);
+ nested_function->reset(nested_place(place));
+ }
+
+ bool has_trivial_destructor() const override {
+ return nested_function->has_trivial_destructor();
+ }
+
+ size_t size_of_data() const override { return prefix_size + nested_function->size_of_data(); }
+
+ size_t align_of_data() const override { return nested_function->align_of_data(); }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena* arena) const override {
+ if (result_is_nullable && get_flag(rhs)) set_flag(place);
+
+ nested_function->merge(nested_place(place), nested_place(rhs), arena);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ bool flag = get_flag(place);
+ if (result_is_nullable) write_binary(flag, buf);
+ if (flag) {
+ nested_function->serialize(nested_place(place), buf);
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena* arena) const override {
+ bool flag = true;
+ if (result_is_nullable) read_binary(flag, buf);
+ if (flag) {
+ set_flag(place);
+ nested_function->deserialize(nested_place(place), buf, arena);
+ }
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ if constexpr (result_is_nullable) {
+ ColumnNullable& to_concrete = assert_cast<ColumnNullable&>(to);
+ if (get_flag(place)) {
+ nested_function->insert_result_into(nested_place(place),
+ to_concrete.get_nested_column());
+ to_concrete.get_null_map_data().push_back(0);
+ } else {
+ to_concrete.insert_default();
+ }
+ } else {
+ nested_function->insert_result_into(nested_place(place), to);
+ }
+ }
+
+ bool allocates_memory_in_arena() const override {
+ return nested_function->allocates_memory_in_arena();
+ }
+
+ bool is_state() const override { return nested_function->is_state(); }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+/** There are two cases: for single argument and variadic.
+ * Code for single argument is much more efficient.
+ */
+template <bool result_is_nullable>
+class AggregateFunctionNullUnary final
+ : public AggregateFunctionNullBase<result_is_nullable,
+ AggregateFunctionNullUnary<result_is_nullable>> {
+public:
+ AggregateFunctionNullUnary(AggregateFunctionPtr nested_function_, const DataTypes& arguments,
+ const Array& params)
+ : AggregateFunctionNullBase<result_is_nullable,
+ AggregateFunctionNullUnary<result_is_nullable>>(
+ std::move(nested_function_), arguments, params) {}
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena* arena) const override {
+ const ColumnNullable* column = assert_cast<const ColumnNullable*>(columns[0]);
+ if (!column->is_null_at(row_num)) {
+ this->set_flag(place);
+ const IColumn* nested_column = &column->get_nested_column();
+ this->nested_function->add(this->nested_place(place), &nested_column, row_num, arena);
+ }
+ }
+
+ void add_batch_single_place(size_t batch_size, AggregateDataPtr place, const IColumn** columns,
+ Arena* arena) const override {
+ const ColumnNullable* column = assert_cast<const ColumnNullable*>(columns[0]);
+ bool has_null = column->has_null();
+
+ if (has_null) {
+ for (size_t i = 0; i < batch_size; ++i) {
+ if (!column->is_null_at(i)) {
+ this->set_flag(place);
+ this->add(place, columns, i, arena);
+ }
+ }
+ } else {
+ this->set_flag(place);
+ const IColumn* nested_column = &column->get_nested_column();
+ this->nested_function->add_batch_single_place(batch_size, this->nested_place(place),
+ &nested_column, arena);
+ }
+ }
+
+ void add_batch_range(size_t batch_begin, size_t batch_end, AggregateDataPtr place,
+ const IColumn** columns, Arena* arena, bool has_null) override {
+ const ColumnNullable* column = assert_cast<const ColumnNullable*>(columns[0]);
+
+ if (has_null) {
+ for (size_t i = batch_begin; i <= batch_end; ++i) {
+ if (!column->is_null_at(i)) {
+ this->set_flag(place);
+ this->add(place, columns, i, arena);
+ }
+ }
+ } else {
+ this->set_flag(place);
+ const IColumn* nested_column = &column->get_nested_column();
+ this->nested_function->add_batch_range(
+ batch_begin, batch_end, this->nested_place(place), &nested_column, arena);
+ }
+ }
+};
+
+template <bool result_is_nullable>
+class AggregateFunctionNullVariadic final
+ : public AggregateFunctionNullBase<result_is_nullable,
+ AggregateFunctionNullVariadic<result_is_nullable>> {
+public:
+ AggregateFunctionNullVariadic(AggregateFunctionPtr nested_function_, const DataTypes& arguments,
+ const Array& params)
+ : AggregateFunctionNullBase<result_is_nullable,
+ AggregateFunctionNullVariadic<result_is_nullable>>(
+ std::move(nested_function_), arguments, params),
+ number_of_arguments(arguments.size()) {
+ if (number_of_arguments == 1) {
+ LOG(FATAL)
+ << "Logical error: single argument is passed to AggregateFunctionNullVariadic";
+ }
+
+ if (number_of_arguments > MAX_ARGS) {
+ LOG(FATAL) << fmt::format(
+ "Maximum number of arguments for aggregate function with Nullable types is {}",
+ size_t(MAX_ARGS));
+ }
+
+ for (size_t i = 0; i < number_of_arguments; ++i)
+ is_nullable[i] = arguments[i]->is_nullable();
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena* arena) const override {
+ /// This container stores the columns we really pass to the nested function.
+ const IColumn* nested_columns[number_of_arguments];
+
+ for (size_t i = 0; i < number_of_arguments; ++i) {
+ if (is_nullable[i]) {
+ const ColumnNullable& nullable_col =
+ assert_cast<const ColumnNullable&>(*columns[i]);
+ if (nullable_col.is_null_at(row_num)) {
+ /// If at least one column has a null value in the current row,
+ /// we don't process this row.
+ return;
+ }
+ nested_columns[i] = &nullable_col.get_nested_column();
+ } else
+ nested_columns[i] = columns[i];
+ }
+
+ this->set_flag(place);
+ this->nested_function->add(this->nested_place(place), nested_columns, row_num, arena);
+ }
+
+ bool allocates_memory_in_arena() const override {
+ return this->nested_function->allocates_memory_in_arena();
+ }
+
+private:
+ enum { MAX_ARGS = 8 };
+ size_t number_of_arguments = 0;
+ std::array<char, MAX_ARGS>
+ is_nullable; /// Plain array is better than std::vector due to one indirection less.
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_reader.cpp b/be/src/vec/aggregate_functions/aggregate_function_reader.cpp
new file mode 100644
index 0000000..9a24ac5
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_reader.cpp
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/aggregate_functions/aggregate_function_reader.h"
+
+namespace doris::vectorized {
+
+// auto spread at nullable condition, null value do not participate aggregate
+void register_aggregate_function_reader(AggregateFunctionSimpleFactory& factory) {
+ // add a suffix to the function name here to distinguish special functions of agg reader
+ auto register_function_reader = [&](const std::string& name,
+ const AggregateFunctionCreator& creator,
+ bool nullable = false) {
+ factory.register_function(name + agg_reader_suffix, creator, nullable);
+ };
+
+ register_function_reader("sum", create_aggregate_function_sum_reader);
+ register_function_reader("max", create_aggregate_function_max);
+ register_function_reader("min", create_aggregate_function_min);
+ register_function_reader("replace_if_not_null", create_aggregate_function_replace_if_not_null);
+ register_function_reader("bitmap_union", create_aggregate_function_bitmap_union);
+ register_function_reader("hll_union", create_aggregate_function_HLL_union);
+}
+
+void register_aggregate_function_reader_no_spread(AggregateFunctionSimpleFactory& factory) {
+ auto register_function_reader = [&](const std::string& name,
+ const AggregateFunctionCreator& creator,
+ bool nullable = false) {
+ factory.register_function(name + agg_reader_suffix, creator, nullable);
+ };
+
+ register_function_reader("replace", create_aggregate_function_replace, false);
+ register_function_reader("replace", create_aggregate_function_replace_nullable, true);
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/olap/rowset/segment_v2/empty_segment_iterator.h b/be/src/vec/aggregate_functions/aggregate_function_reader.h
similarity index 54%
copy from be/src/olap/rowset/segment_v2/empty_segment_iterator.h
copy to be/src/vec/aggregate_functions/aggregate_function_reader.h
index cb6c48c..f44be5e 100644
--- a/be/src/olap/rowset/segment_v2/empty_segment_iterator.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_reader.h
@@ -17,25 +17,19 @@
#pragma once
-#include "common/status.h"
-#include "olap/iterators.h"
-#include "olap/rowset/segment_v2/segment.h"
-#include "olap/schema.h"
+#include "vec/aggregate_functions/aggregate_function_bitmap.h"
+#include "vec/aggregate_functions/aggregate_function_hll_union_agg.h"
+#include "vec/aggregate_functions/aggregate_function_min_max.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/aggregate_function_sum.h"
+#include "vec/aggregate_functions/aggregate_function_window.h"
-namespace doris {
-namespace segment_v2 {
+namespace doris::vectorized {
-class EmptySegmentIterator : public RowwiseIterator {
-public:
- explicit EmptySegmentIterator(const Schema& schema);
- ~EmptySegmentIterator() override {}
- Status init(const StorageReadOptions& opts) override { return Status::OK(); }
- const Schema& schema() const override { return _schema; }
- Status next_batch(RowBlockV2* row_block) override;
+static const std::string agg_reader_suffix = "_reader";
-private:
- Schema _schema;
-};
+void register_aggregate_function_reader(AggregateFunctionSimpleFactory& factory);
-} // namespace segment_v2
-} // namespace doris
\ No newline at end of file
+void register_aggregate_function_reader_no_spread(AggregateFunctionSimpleFactory& factory);
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
new file mode 100644
index 0000000..8a1995b
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionFactory.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+
+#include "vec/aggregate_functions/aggregate_function_reader.h"
+
+namespace doris::vectorized {
+class AggregateFunctionSimpleFactory;
+void register_aggregate_function_sum(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_combinator_null(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_minmax(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_avg(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_count(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_HLL_union_agg(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_uniq(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_combinator_distinct(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_bitmap(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_window_rank(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_window_lead_lag(AggregateFunctionSimpleFactory& factory);
+AggregateFunctionSimpleFactory& AggregateFunctionSimpleFactory::instance() {
+ static std::once_flag oc;
+ static AggregateFunctionSimpleFactory instance;
+ std::call_once(oc, [&]() {
+ register_aggregate_function_sum(instance);
+ register_aggregate_function_minmax(instance);
+ register_aggregate_function_avg(instance);
+ register_aggregate_function_count(instance);
+ register_aggregate_function_uniq(instance);
+ register_aggregate_function_bitmap(instance);
+ register_aggregate_function_combinator_distinct(instance);
+ register_aggregate_function_HLL_union_agg(instance);
+ register_aggregate_function_reader(instance); // register aggregate function for agg reader
+ register_aggregate_function_window_rank(instance);
+
+ // if you only register function with no nullable, and wants to add nullable automatically, you should place function above this line
+ register_aggregate_function_combinator_null(instance);
+
+ register_aggregate_function_reader_no_spread(instance);
+ register_aggregate_function_window_lead_lag(instance);
+ });
+ return instance;
+}
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h
new file mode 100644
index 0000000..d12c27a
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionFactory.h
+// and modified by Doris
+
+#pragma once
+
+#include <functional>
+#include <iostream>
+#include <memory>
+#include <mutex>
+#include <unordered_map>
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/core/field.h"
+#include "vec/data_types/data_type.h"
+
+namespace doris::vectorized {
+using DataTypePtr = std::shared_ptr<const IDataType>;
+using DataTypes = std::vector<DataTypePtr>;
+using AggregateFunctionCreator = std::function<AggregateFunctionPtr(
+ const std::string&, const DataTypes&, const Array&, const bool)>;
+
+class AggregateFunctionSimpleFactory {
+public:
+ using Creator = AggregateFunctionCreator;
+
+private:
+ using AggregateFunctions = std::unordered_map<std::string, Creator>;
+
+ AggregateFunctions aggregate_functions;
+ AggregateFunctions nullable_aggregate_functions;
+
+public:
+ void register_nullable_function_combinator(const Creator& creator) {
+ for (const auto& entity : aggregate_functions) {
+ if (nullable_aggregate_functions.find(entity.first) ==
+ nullable_aggregate_functions.end()) {
+ nullable_aggregate_functions[entity.first] = creator;
+ }
+ }
+ }
+
+ void register_distinct_function_combinator(const Creator& creator, const std::string& prefix) {
+ std::vector<std::string> need_insert;
+ for (const auto& entity : aggregate_functions) {
+ std::string target_value = prefix + entity.first;
+ if (aggregate_functions.find(target_value) == aggregate_functions.end()) {
+ need_insert.emplace_back(std::move(target_value));
+ }
+ }
+ for (const auto& function_name : need_insert) {
+ aggregate_functions[function_name] = creator;
+ }
+ }
+
+ AggregateFunctionPtr get(const std::string& name, const DataTypes& argument_types,
+ const Array& parameters, const bool result_is_nullable = false) {
+ bool nullable = false;
+ for (const auto& type : argument_types) {
+ if (type->is_nullable()) {
+ nullable = true;
+ }
+ }
+ if (nullable) {
+ return nullable_aggregate_functions.find(name) == nullable_aggregate_functions.end()
+ ? nullptr
+ : nullable_aggregate_functions[name](name, argument_types, parameters,
+ result_is_nullable);
+ } else {
+ return aggregate_functions.find(name) == aggregate_functions.end()
+ ? nullptr
+ : aggregate_functions[name](name, argument_types, parameters,
+ result_is_nullable);
+ }
+ }
+
+ void register_function(const std::string& name, const Creator& creator, bool nullable = false) {
+ if (nullable) {
+ nullable_aggregate_functions[name] = creator;
+ } else {
+ aggregate_functions[name] = creator;
+ }
+ }
+
+public:
+ static AggregateFunctionSimpleFactory& instance();
+};
+}; // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_sum.cpp b/be/src/vec/aggregate_functions/aggregate_function_sum.cpp
new file mode 100644
index 0000000..f7127e7
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_sum.cpp
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionSum.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_sum.h"
+
+#include <fmt/format.h>
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/helpers.h"
+
+namespace doris::vectorized {
+
+template <typename T>
+struct SumSimple {
+ /// @note It uses slow Decimal128 (cause we need such a variant). sumWithOverflow is faster for Decimal32/64
+ using ResultType = std::conditional_t<IsDecimalNumber<T>, Decimal128, NearestFieldType<T>>;
+ // using ResultType = NearestFieldType<T>;
+ using AggregateDataType = AggregateFunctionSumData<ResultType>;
+ using Function = AggregateFunctionSum<T, ResultType, AggregateDataType>;
+};
+
+template <typename T>
+using AggregateFunctionSumSimple = typename SumSimple<T>::Function;
+
+template <template <typename> class Function>
+AggregateFunctionPtr create_aggregate_function_sum(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ // assert_no_parameters(name, parameters);
+ // assert_unary(name, argument_types);
+
+ AggregateFunctionPtr res;
+ DataTypePtr data_type = argument_types[0];
+ if (is_decimal(data_type))
+ res.reset(create_with_decimal_type<Function>(*data_type, *data_type, argument_types));
+ else
+ res.reset(create_with_numeric_type<Function>(*data_type, argument_types));
+
+ if (!res) {
+ LOG(WARNING) << fmt::format("Illegal type {} of argument for aggregate function {}",
+ argument_types[0]->get_name(), name);
+ }
+ return res;
+}
+
+// do not level up return type for agg reader
+template <typename T>
+struct SumSimpleReader {
+ using ResultType = T;
+ using AggregateDataType = AggregateFunctionSumData<ResultType>;
+ using Function = AggregateFunctionSum<T, ResultType, AggregateDataType>;
+};
+
+template <typename T>
+using AggregateFunctionSumSimpleReader = typename SumSimpleReader<T>::Function;
+
+AggregateFunctionPtr create_aggregate_function_sum_reader(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return create_aggregate_function_sum<AggregateFunctionSumSimpleReader>(
+ name, argument_types, parameters, result_is_nullable);
+}
+
+void register_aggregate_function_sum(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("sum", create_aggregate_function_sum<AggregateFunctionSumSimple>);
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_sum.h b/be/src/vec/aggregate_functions/aggregate_function_sum.h
new file mode 100644
index 0000000..402af36
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_sum.h
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionSum.h
+// and modified by Doris
+
+#pragma once
+
+#include <istream>
+#include <ostream>
+#include <type_traits>
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/io/io_helper.h"
+
+namespace doris::vectorized {
+
+template <typename T>
+struct AggregateFunctionSumData {
+ T sum {};
+
+ void add(T value) { sum += value; }
+
+ void merge(const AggregateFunctionSumData& rhs) { sum += rhs.sum; }
+
+ void write(BufferWritable& buf) const { write_binary(sum, buf); }
+
+ void read(BufferReadable& buf) { read_binary(sum, buf); }
+
+ T get() const { return sum; }
+};
+
+/// Counts the sum of the numbers.
+template <typename T, typename TResult, typename Data>
+class AggregateFunctionSum final
+ : public IAggregateFunctionDataHelper<Data, AggregateFunctionSum<T, TResult, Data>> {
+public:
+ using ResultDataType = std::conditional_t<IsDecimalNumber<T>, DataTypeDecimal<TResult>,
+ DataTypeNumber<TResult>>;
+ using ColVecType = std::conditional_t<IsDecimalNumber<T>, ColumnDecimal<T>, ColumnVector<T>>;
+ using ColVecResult =
+ std::conditional_t<IsDecimalNumber<T>, ColumnDecimal<TResult>, ColumnVector<TResult>>;
+
+ String get_name() const override { return "sum"; }
+
+ AggregateFunctionSum(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionSum<T, TResult, Data>>(
+ argument_types_, {}),
+ scale(0) {}
+
+ AggregateFunctionSum(const IDataType& data_type, const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionSum<T, TResult, Data>>(
+ argument_types_, {}),
+ scale(get_decimal_scale(data_type)) {}
+
+ DataTypePtr get_return_type() const override {
+ if constexpr (IsDecimalNumber<T>)
+ return std::make_shared<ResultDataType>(ResultDataType::max_precision(), scale);
+ else
+ return std::make_shared<ResultDataType>();
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ const auto& column = static_cast<const ColVecType&>(*columns[0]);
+ this->data(place).add(column.get_data()[row_num]);
+ }
+
+ void reset(AggregateDataPtr place) const override {
+ this->data(place).sum = {};
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ this->data(place).merge(this->data(rhs));
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ auto& column = static_cast<ColVecResult&>(to);
+ column.get_data().push_back(this->data(place).get());
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+
+private:
+ UInt32 scale;
+};
+
+AggregateFunctionPtr create_aggregate_function_sum_reader(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable);
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_uniq.cpp b/be/src/vec/aggregate_functions/aggregate_function_uniq.cpp
new file mode 100644
index 0000000..0258858
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_uniq.cpp
@@ -0,0 +1,69 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionUniq.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_uniq.h"
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+#include "vec/aggregate_functions/helpers.h"
+#include "vec/data_types/data_type_string.h"
+
+namespace doris::vectorized {
+
+template <template <typename> class Data, typename DataForVariadic>
+AggregateFunctionPtr create_aggregate_function_uniq(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& params,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, params);
+
+ if (argument_types.empty()) {
+ LOG(WARNING) << "Incorrect number of arguments for aggregate function " << name;
+ return nullptr;
+ }
+
+ if (argument_types.size() == 1) {
+ const IDataType& argument_type = *argument_types[0];
+
+ AggregateFunctionPtr res(create_with_numeric_type<AggregateFunctionUniq, Data>(
+ *argument_types[0], argument_types));
+
+ WhichDataType which(argument_type);
+ // TODO: DateType
+ if (res)
+ return res;
+ else if (which.is_decimal())
+ return std::make_shared<AggregateFunctionUniq<Decimal128, Data<Int128>>>(argument_types);
+ else if (which.is_string_or_fixed_string())
+ return std::make_shared<AggregateFunctionUniq<String, Data<String>>>(argument_types);
+ }
+
+ return nullptr;
+}
+
+void register_aggregate_function_uniq(AggregateFunctionSimpleFactory& factory) {
+ AggregateFunctionCreator creator =
+ create_aggregate_function_uniq<AggregateFunctionUniqExactData,
+ AggregateFunctionUniqExactData<String>>;
+ factory.register_function("multi_distinct_count", creator);
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_uniq.h b/be/src/vec/aggregate_functions/aggregate_function_uniq.h
new file mode 100644
index 0000000..ea32404
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_uniq.h
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/AggregateFunctions/AggregateFunctionUniq.h
+// and modified by Doris
+
+#pragma once
+
+#include <type_traits>
+
+#include "gutil/hash/city.h"
+
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/common/aggregation_common.h"
+#include "vec/common/assert_cast.h"
+#include "vec/common/bit_cast.h"
+#include "vec/common/hash_table/hash_set.h"
+#include "vec/common/typeid_cast.h"
+#include "vec/data_types/data_type_number.h"
+
+namespace doris::vectorized {
+
+/// uniqExact
+
+template <typename T>
+struct AggregateFunctionUniqExactData {
+ using Key = T;
+
+ /// When creating, the hash table must be small.
+ using Set = HashSet<Key, HashCRC32<Key>, HashTableGrower<4>,
+ HashTableAllocatorWithStackMemory<sizeof(Key) * (1 << 4)>>;
+
+ Set set;
+
+ static String get_name() { return "uniqExact"; }
+};
+
+/// For rows, we put the SipHash values (128 bits) into the hash table.
+template <>
+struct AggregateFunctionUniqExactData<String> {
+ using Key = UInt128;
+
+ /// When creating, the hash table must be small.
+ using Set = HashSet<Key, UInt128TrivialHash, HashTableGrower<3>,
+ HashTableAllocatorWithStackMemory<sizeof(Key) * (1 << 3)>>;
+
+ Set set;
+
+ static String get_name() { return "uniqExact"; }
+};
+
+namespace detail {
+
+/** The structure for the delegation work to add one element to the `uniq` aggregate functions.
+ * Used for partial specialization to add strings.
+ */
+template <typename T, typename Data>
+struct OneAdder {
+ static void ALWAYS_INLINE add(Data& data, const IColumn& column, size_t row_num) {
+ if constexpr (std::is_same_v<T, String>) {
+ StringRef value = column.get_data_at(row_num);
+
+ UInt128 key;
+ SipHash hash;
+ hash.update(value.data, value.size);
+ hash.get128(key.low, key.high);
+
+ data.set.insert(key);
+ } else if constexpr(std::is_same_v<T, Decimal128>) {
+ data.set.insert(assert_cast<const ColumnDecimal<Decimal128>&>(column).get_data()[row_num]);
+ } else {
+ data.set.insert(assert_cast<const ColumnVector<T>&>(column).get_data()[row_num]);
+ }
+ }
+};
+
+} // namespace detail
+
+/// Calculates the number of different values approximately or exactly.
+template <typename T, typename Data>
+class AggregateFunctionUniq final
+ : public IAggregateFunctionDataHelper<Data, AggregateFunctionUniq<T, Data>> {
+public:
+ AggregateFunctionUniq(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionUniq<T, Data>>(argument_types_,
+ {}) {}
+
+ String get_name() const override { return Data::get_name(); }
+
+ DataTypePtr get_return_type() const override { return std::make_shared<DataTypeInt64>(); }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ detail::OneAdder<T, Data>::add(this->data(place), *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena*) const override {
+ this->data(place).set.merge(this->data(rhs).set);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).set.write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf, Arena*) const override {
+ this->data(place).set.read(buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ assert_cast<ColumnInt64&>(to).get_data().push_back(this->data(place).set.size());
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_window.cpp b/be/src/vec/aggregate_functions/aggregate_function_window.cpp
new file mode 100644
index 0000000..c40f2bc
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_window.cpp
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Processors/Transforms/WindowTransform.cpp
+// and modified by Doris
+
+#include "vec/aggregate_functions/aggregate_function_window.h"
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+#include "vec/aggregate_functions/helpers.h"
+namespace doris::vectorized {
+
+AggregateFunctionPtr create_aggregate_function_dense_rank(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+
+ return std::make_shared<WindowFunctionDenseRank>(argument_types);
+}
+
+AggregateFunctionPtr create_aggregate_function_rank(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+
+ return std::make_shared<WindowFunctionRank>(argument_types);
+}
+
+AggregateFunctionPtr create_aggregate_function_row_number(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ assert_no_parameters(name, parameters);
+
+ return std::make_shared<WindowFunctionRowNumber>(argument_types);
+}
+
+template <template <typename> class AggregateFunctionTemplate, template <typename> class Data,
+ bool is_nullable, bool is_copy = false>
+static IAggregateFunction* create_function_single_value(const String& name,
+ const DataTypes& argument_types,
+ const Array& parameters) {
+ using StoreType = std::conditional_t<is_copy, CopiedValue, Value>;
+
+ assert_arity_at_most<3>(name, argument_types);
+
+ auto type = argument_types[0].get();
+ if (type->is_nullable()) {
+ type = assert_cast<const DataTypeNullable*>(type)->get_nested_type().get();
+ }
+ WhichDataType which(*type);
+
+#define DISPATCH(TYPE) \
+ if (which.idx == TypeIndex::TYPE) \
+ return new AggregateFunctionTemplate< \
+ Data<LeadAndLagData<TYPE, is_nullable, false, StoreType>>>(argument_types);
+ FOR_NUMERIC_TYPES(DISPATCH)
+#undef DISPATCH
+
+ if (which.is_decimal()) {
+ return new AggregateFunctionTemplate<
+ Data<LeadAndLagData<Int128, is_nullable, false, StoreType>>>(argument_types);
+ }
+ if (which.is_date_or_datetime()) {
+ return new AggregateFunctionTemplate<
+ Data<LeadAndLagData<Int64, is_nullable, false, StoreType>>>(argument_types);
+ }
+ if (which.is_string_or_fixed_string())
+ return new AggregateFunctionTemplate<
+ Data<LeadAndLagData<StringRef, is_nullable, true, StoreType>>>(argument_types);
+ DCHECK(false) << "with unknowed type, failed in create_aggregate_function_leadlag";
+ return nullptr;
+}
+
+template <bool is_nullable>
+AggregateFunctionPtr create_aggregate_function_lag(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionLagData, is_nullable>(
+ name, argument_types, parameters));
+}
+
+template <bool is_nullable>
+AggregateFunctionPtr create_aggregate_function_lead(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionLeadData, is_nullable>(
+ name, argument_types, parameters));
+}
+
+template <bool is_nullable>
+AggregateFunctionPtr create_aggregate_function_first(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionFirstData, is_nullable>(
+ name, argument_types, parameters));
+}
+
+template <bool is_nullable>
+AggregateFunctionPtr create_aggregate_function_last(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionLastData, is_nullable>(
+ name, argument_types, parameters));
+}
+
+AggregateFunctionPtr create_aggregate_function_replace_if_not_null(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionFirstData, false, true>(
+ name, argument_types, parameters));
+}
+
+AggregateFunctionPtr create_aggregate_function_replace(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionFirstData, false, true>(
+ name, argument_types, parameters));
+}
+
+AggregateFunctionPtr create_aggregate_function_replace_nullable(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<WindowFunctionData, WindowFunctionFirstData, true, true>(
+ name, argument_types, parameters));
+}
+
+void register_aggregate_function_window_rank(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("dense_rank", create_aggregate_function_dense_rank);
+ factory.register_function("rank", create_aggregate_function_rank);
+ factory.register_function("row_number", create_aggregate_function_row_number);
+}
+
... 62300 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 09/33: [Vectorized][Feature] fix core dump when using function override and function alias at the same time && support substr(str, int) override (#7640)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 41b165cbf10b39ec41f63958e2332aa1d5b8fc5d
Author: Pxl <95...@qq.com>
AuthorDate: Thu Jan 6 18:57:05 2022 +0800
[Vectorized][Feature] fix core dump when using function override and function alias at the same time && support substr(str,int) override (#7640)
---
be/src/vec/functions/function_string.cpp | 19 +++--
be/src/vec/functions/function_string.h | 109 ++++++++++++++++++++-----
be/src/vec/functions/function_timestamp.cpp | 11 +--
be/src/vec/functions/simple_function_factory.h | 10 ++-
gensrc/script/doris_builtins_functions.py | 4 +-
5 files changed, 111 insertions(+), 42 deletions(-)
diff --git a/be/src/vec/functions/function_string.cpp b/be/src/vec/functions/function_string.cpp
index 73e2413..34f1c6b 100644
--- a/be/src/vec/functions/function_string.cpp
+++ b/be/src/vec/functions/function_string.cpp
@@ -293,7 +293,7 @@ struct HexStringImpl {
dst_data_ptr++;
offset++;
} else {
- VStringFunctions::hex_encode(source, srclen, reinterpret_cast<char *>(dst_data_ptr));
+ VStringFunctions::hex_encode(source, srclen, reinterpret_cast<char*>(dst_data_ptr));
dst_data_ptr[srclen * 2] = '\0';
dst_data_ptr += (srclen * 2 + 1);
offset += (srclen * 2 + 1);
@@ -513,9 +513,9 @@ struct AesEncryptImpl {
int cipher_len = l_size + 16;
char p[cipher_len];
- int outlen =
- EncryptionUtil::encrypt(AES_128_ECB, (unsigned char*)l_raw, l_size,
- (unsigned char*)r_raw, r_size, NULL, true, (unsigned char*)p);
+ int outlen = EncryptionUtil::encrypt(AES_128_ECB, (unsigned char*)l_raw, l_size,
+ (unsigned char*)r_raw, r_size, NULL, true,
+ (unsigned char*)p);
if (outlen < 0) {
StringOP::push_null_string(i, res_data, res_offsets, null_map_data);
} else {
@@ -553,9 +553,9 @@ struct AesDecryptImpl {
int cipher_len = l_size;
char p[cipher_len];
- int outlen =
- EncryptionUtil::decrypt(AES_128_ECB, (unsigned char*)l_raw, l_size,
- (unsigned char*)r_raw, r_size, NULL, true, (unsigned char*)p);
+ int outlen = EncryptionUtil::decrypt(AES_128_ECB, (unsigned char*)l_raw, l_size,
+ (unsigned char*)r_raw, r_size, NULL, true,
+ (unsigned char*)p);
if (outlen < 0) {
StringOP::push_null_string(i, res_data, res_offsets, null_map_data);
} else {
@@ -774,7 +774,8 @@ void register_function_string(SimpleFunctionFactory& factory) {
factory.register_function<FunctionLTrim>();
factory.register_function<FunctionRTrim>();
factory.register_function<FunctionTrim>();
- factory.register_function<FunctionSubstring>();
+ factory.register_function<FunctionSubstring<Substr3Imp>>();
+ factory.register_function<FunctionSubstring<Substr2Imp>>();
factory.register_function<FunctionLeft>();
factory.register_function<FunctionRight>();
factory.register_function<FunctionNullOrEmpty>();
@@ -794,7 +795,7 @@ void register_function_string(SimpleFunctionFactory& factory) {
factory.register_alias(FunctionLeft::name, "strleft");
factory.register_alias(FunctionRight::name, "strright");
- factory.register_alias(FunctionSubstring::name, "substr");
+ factory.register_alias(SubstringUtil::name, "substr");
factory.register_alias(FunctionToLower::name, "lcase");
factory.register_alias(FunctionStringMd5sum::name, "md5");
}
diff --git a/be/src/vec/functions/function_string.h b/be/src/vec/functions/function_string.h
index efeef41..3f3e538 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -88,25 +88,9 @@ struct StringOP {
}
};
-class FunctionSubstring : public IFunction {
-public:
+struct SubstringUtil {
static constexpr auto name = "substring";
- static FunctionPtr create() { return std::make_shared<FunctionSubstring>(); }
- String get_name() const override { return name; }
- size_t get_number_of_arguments() const override { return 3; }
-
- DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
- return make_nullable(std::make_shared<DataTypeString>());
- }
-
- bool use_default_implementation_for_nulls() const override { return false; }
- bool use_default_implementation_for_constants() const override { return true; }
- Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
- size_t result, size_t input_rows_count) override {
- substring_execute(block, arguments, result, input_rows_count);
- return Status::OK();
- }
static void substring_execute(Block& block, const ColumnNumbers& arguments, size_t result,
size_t input_rows_count) {
DCHECK_EQ(arguments.size(), 3);
@@ -201,12 +185,89 @@ private:
}
};
-class FunctionLeft : public FunctionSubstring {
+template <typename Impl>
+class FunctionSubstring : public IFunction {
+public:
+ static constexpr auto name = SubstringUtil::name;
+ String get_name() const override { return name; }
+ static FunctionPtr create() { return std::make_shared<FunctionSubstring<Impl>>(); }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+ DataTypes get_variadic_argument_types_impl() const override {
+ return Impl::get_variadic_argument_types();
+ }
+ size_t get_number_of_arguments() const override {
+ return get_variadic_argument_types_impl().size();
+ }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+ bool use_default_implementation_for_constants() const override { return true; }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ return Impl::execute_impl(context, block, arguments, result, input_rows_count);
+ }
+};
+
+struct Substr3Imp {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeInt32>(),
+ std::make_shared<DataTypeInt32>()};
+ }
+
+ static Status execute_impl(FunctionContext* context, Block& block,
+ const ColumnNumbers& arguments, size_t result,
+ size_t input_rows_count) {
+ SubstringUtil::substring_execute(block, arguments, result, input_rows_count);
+ return Status::OK();
+ }
+};
+
+struct Substr2Imp {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeInt32>()};
+ }
+
+ static Status execute_impl(FunctionContext* context, Block& block,
+ const ColumnNumbers& arguments, size_t result,
+ size_t input_rows_count) {
+ auto params = ColumnInt32::create(input_rows_count);
+ auto& strlen_data = params->get_data();
+
+ auto str_col =
+ block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+ if (auto* nullable = check_and_get_column<const ColumnNullable>(*str_col)) {
+ str_col = nullable->get_nested_column_ptr();
+ }
+ auto& str_offset = assert_cast<const ColumnString*>(str_col.get())->get_offsets();
+
+ for (int i = 0; i < input_rows_count; ++i) {
+ strlen_data[i] = str_offset[i] - str_offset[i - 1];
+ }
+
+ block.insert({std::move(params), std::make_shared<DataTypeInt32>(), "strlen"});
+
+ ColumnNumbers temp_arguments = {arguments[0], arguments[1], block.columns() - 1};
+
+ SubstringUtil::substring_execute(block, temp_arguments, result, input_rows_count);
+ return Status::OK();
+ }
+};
+
+class FunctionLeft : public IFunction {
public:
static constexpr auto name = "left";
static FunctionPtr create() { return std::make_shared<FunctionLeft>(); }
String get_name() const override { return name; }
size_t get_number_of_arguments() const override { return 2; }
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+ bool use_default_implementation_for_constants() const override { return true; }
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) override {
@@ -219,17 +280,23 @@ public:
temp_arguments[0] = arguments[0];
temp_arguments[1] = num_columns_without_result;
temp_arguments[2] = arguments[1];
- FunctionSubstring::substring_execute(block, temp_arguments, result, input_rows_count);
+ SubstringUtil::substring_execute(block, temp_arguments, result, input_rows_count);
return Status::OK();
}
};
-class FunctionRight : public FunctionSubstring {
+class FunctionRight : public IFunction {
public:
static constexpr auto name = "right";
static FunctionPtr create() { return std::make_shared<FunctionRight>(); }
String get_name() const override { return name; }
size_t get_number_of_arguments() const override { return 2; }
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+ bool use_default_implementation_for_constants() const override { return true; }
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) override {
@@ -275,7 +342,7 @@ public:
temp_arguments[0] = arguments[0];
temp_arguments[1] = num_columns_without_result;
temp_arguments[2] = num_columns_without_result + 1;
- FunctionSubstring::substring_execute(block, temp_arguments, result, input_rows_count);
+ SubstringUtil::substring_execute(block, temp_arguments, result, input_rows_count);
return Status::OK();
}
};
diff --git a/be/src/vec/functions/function_timestamp.cpp b/be/src/vec/functions/function_timestamp.cpp
index c7335c4..f5216e9 100644
--- a/be/src/vec/functions/function_timestamp.cpp
+++ b/be/src/vec/functions/function_timestamp.cpp
@@ -177,9 +177,7 @@ struct UnixTimeStampImpl {
};
struct UnixTimeStampDateImpl {
- static DataTypes get_variadic_argument_types() {
- return {std::make_shared<vectorized::DataTypeDate>()};
- }
+ static DataTypes get_variadic_argument_types() { return {std::make_shared<DataTypeDate>()}; }
static DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments) {
return make_nullable(std::make_shared<DataTypeInt32>());
@@ -226,14 +224,13 @@ struct UnixTimeStampDateImpl {
struct UnixTimeStampDatetimeImpl : public UnixTimeStampDateImpl {
static DataTypes get_variadic_argument_types() {
- return {std::make_shared<vectorized::DataTypeDateTime>()};
+ return {std::make_shared<DataTypeDateTime>()};
}
};
struct UnixTimeStampStrImpl {
static DataTypes get_variadic_argument_types() {
- return {std::make_shared<vectorized::DataTypeString>(),
- std::make_shared<vectorized::DataTypeString>()};
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>()};
}
static DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments) {
@@ -294,8 +291,6 @@ public:
String get_name() const override { return name; }
- bool is_variadic() const override { return true; }
-
bool use_default_implementation_for_nulls() const override { return false; }
size_t get_number_of_arguments() const override {
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index 149ce88..c016d35 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -95,14 +95,19 @@ public:
}
void register_alias(const std::string& name, const std::string& alias) {
- function_creators[alias] = function_creators[name];
+ function_alias[alias] = name;
}
FunctionBasePtr get_function(const std::string& name, const ColumnsWithTypeAndName& arguments,
const DataTypePtr& return_type) {
std::string key_str = name;
+
+ if (function_alias.count(name)) {
+ key_str = function_alias[name];
+ }
+
// if function is variadic, added types_str as key
- if (function_variadic_set.count(name)) {
+ if (function_variadic_set.count(key_str)) {
for (auto& arg : arguments) {
key_str.append(arg.type->is_nullable()
? reinterpret_cast<const DataTypeNullable*>(arg.type.get())
@@ -123,6 +128,7 @@ public:
private:
FunctionCreators function_creators;
FunctionIsVariadic function_variadic_set;
+ std::unordered_map<std::string, std::string> function_alias;
template <typename Function>
static FunctionBuilderPtr createDefaultFunction() {
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 7bf7d2d..5027627 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -828,7 +828,7 @@ visible_functions = [
# String builtin functions
[['substr', 'substring'], 'VARCHAR', ['VARCHAR', 'INT'],
'_ZN5doris15StringFunctions9substringEPN'
- '9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', '', 'ALWAYS_NULLABLE'],
+ '9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['substr', 'substring'], 'VARCHAR', ['VARCHAR', 'INT', 'INT'],
'_ZN5doris15StringFunctions9substringEPN'
'9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValES9_', '', '', 'vec', 'ALWAYS_NULLABLE'],
@@ -950,7 +950,7 @@ visible_functions = [
# Longtext function
[['substr', 'substring'], 'STRING', ['STRING', 'INT'],
'_ZN5doris15StringFunctions9substringEPN'
- '9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', '', 'ALWAYS_NULLABLE'],
+ '9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValE', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['substr', 'substring'], 'STRING', ['STRING', 'INT', 'INT'],
'_ZN5doris15StringFunctions9substringEPN'
'9doris_udf15FunctionContextERKNS1_9StringValERKNS1_6IntValES9_', '', '', 'vec', 'ALWAYS_NULLABLE'],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 24/33: [Vectorized][Bug] Bitmap/HLL type no support cast to varchar/char (#7737)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 30de672b2e45a83a95706a1f1f100054b436f33c
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Thu Jan 13 13:34:29 2022 +0800
[Vectorized][Bug] Bitmap/HLL type no support cast to varchar/char (#7737)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/vec/data_types/data_type_bitmap.cpp | 9 +++++++++
be/src/vec/data_types/data_type_bitmap.h | 1 +
be/src/vec/data_types/data_type_string.cpp | 8 ++++++++
be/src/vec/data_types/data_type_string.h | 1 +
4 files changed, 19 insertions(+)
diff --git a/be/src/vec/data_types/data_type_bitmap.cpp b/be/src/vec/data_types/data_type_bitmap.cpp
index 4daa72e..c6bc9f0 100644
--- a/be/src/vec/data_types/data_type_bitmap.cpp
+++ b/be/src/vec/data_types/data_type_bitmap.cpp
@@ -90,4 +90,13 @@ void DataTypeBitMap::deserialize_as_stream(BitmapValue& value, BufferReadable& b
read_string_binary(ref, buf);
value.deserialize(ref.data);
}
+
+void DataTypeBitMap::to_string(const class doris::vectorized::IColumn& column, size_t row_num,
+ doris::vectorized::BufferWritable& ostr) const {
+ auto& data = const_cast<BitmapValue&>(assert_cast<const ColumnBitmap&>(column).get_element(row_num));
+ std::string result(data.getSizeInBytes(), '0');
+ data.write((char*)result.data());
+
+ ostr.write(result.data(), result.size());
+}
} // namespace doris::vectorized
diff --git a/be/src/vec/data_types/data_type_bitmap.h b/be/src/vec/data_types/data_type_bitmap.h
index 69f5540..c2166fb 100644
--- a/be/src/vec/data_types/data_type_bitmap.h
+++ b/be/src/vec/data_types/data_type_bitmap.h
@@ -65,6 +65,7 @@ public:
bool can_be_inside_low_cardinality() const override { return false; }
std::string to_string(const IColumn& column, size_t row_num) const { return "BitMap()"; }
+ void to_string(const IColumn &column, size_t row_num, BufferWritable &ostr) const override;
[[noreturn]] virtual Field get_default() const {
LOG(FATAL) << "Method get_default() is not implemented for data type " << get_name();
diff --git a/be/src/vec/data_types/data_type_string.cpp b/be/src/vec/data_types/data_type_string.cpp
index f481721..86b0aac 100644
--- a/be/src/vec/data_types/data_type_string.cpp
+++ b/be/src/vec/data_types/data_type_string.cpp
@@ -58,6 +58,14 @@ std::string DataTypeString::to_string(const IColumn& column, size_t row_num) con
return s.to_string();
}
+void DataTypeString::to_string(const class doris::vectorized::IColumn & column, size_t row_num,
+ class doris::vectorized::BufferWritable & ostr) const {
+ const StringRef& s =
+ assert_cast<const ColumnString&>(*column.convert_to_full_column_if_const().get())
+ .get_data_at(row_num);
+ ostr.write(s.data, s.size);
+}
+
Field DataTypeString::get_default() const {
return String();
}
diff --git a/be/src/vec/data_types/data_type_string.h b/be/src/vec/data_types/data_type_string.h
index 9d5b21b..8506473 100644
--- a/be/src/vec/data_types/data_type_string.h
+++ b/be/src/vec/data_types/data_type_string.h
@@ -55,6 +55,7 @@ public:
bool can_be_inside_nullable() const override { return true; }
bool can_be_inside_low_cardinality() const override { return true; }
std::string to_string(const IColumn& column, size_t row_num) const;
+ void to_string(const IColumn &column, size_t row_num, BufferWritable &ostr) const override;
};
} // namespace doris::vectorized
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 23/33: [Vectorized][Feature] upport function conv (#7693)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit ffdc9fc9be28ae94671916f06e4fd5f219707e93
Author: Pxl <95...@qq.com>
AuthorDate: Wed Jan 12 17:08:09 2022 +0800
[Vectorized][Feature] upport function conv (#7693)
* support function conv()
* add document
---
be/src/exprs/math_functions.h | 14 +-
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/data_types/data_type_bitmap.h | 2 +
be/src/vec/data_types/data_type_date.h | 2 +-
be/src/vec/data_types/data_type_date_time.h | 40 ++---
be/src/vec/data_types/data_type_decimal.h | 2 +-
be/src/vec/data_types/data_type_number_base.h | 5 +-
be/src/vec/data_types/data_type_string.h | 5 +-
be/src/vec/functions/function_conv.cpp | 163 +++++++++++++++++++++
be/src/vec/functions/simple_function_factory.h | 2 +
docs/.vuepress/sidebar/en.js | 5 +
docs/.vuepress/sidebar/zh-CN.js | 5 +
.../sql-functions/math-functions/conv.md | 60 ++++++++
.../sql-functions/math-functions/conv.md | 60 ++++++++
gensrc/script/doris_builtins_functions.py | 6 +-
15 files changed, 339 insertions(+), 33 deletions(-)
diff --git a/be/src/exprs/math_functions.h b/be/src/exprs/math_functions.h
index 15d8749..9d55ed6 100644
--- a/be/src/exprs/math_functions.h
+++ b/be/src/exprs/math_functions.h
@@ -50,7 +50,8 @@ public:
static doris_udf::IntVal abs(doris_udf::FunctionContext*, const doris_udf::SmallIntVal&);
static doris_udf::SmallIntVal abs(doris_udf::FunctionContext*, const doris_udf::TinyIntVal&);
- static doris_udf::TinyIntVal sign(doris_udf::FunctionContext* ctx, const doris_udf::DoubleVal& v);
+ static doris_udf::TinyIntVal sign(doris_udf::FunctionContext* ctx,
+ const doris_udf::DoubleVal& v);
static doris_udf::DoubleVal sin(doris_udf::FunctionContext*, const doris_udf::DoubleVal&);
static doris_udf::DoubleVal asin(doris_udf::FunctionContext*, const doris_udf::DoubleVal&);
@@ -182,11 +183,6 @@ public:
static double my_double_round(double value, int64_t dec, bool dec_unsigned, bool truncate);
-private:
- static const int32_t MIN_BASE = 2;
- static const int32_t MAX_BASE = 36;
- static const char* _s_alphanumeric_chars;
-
// Converts src_num in decimal to dest_base,
// and fills expr_val.string_val with the result.
static doris_udf::StringVal decimal_to_base(doris_udf::FunctionContext* ctx, int64_t src_num,
@@ -207,6 +203,12 @@ private:
// Returns false otherwise, indicating some other error condition.
static bool handle_parse_result(int8_t dest_base, int64_t* num,
StringParser::ParseResult parse_res);
+
+ static const int32_t MIN_BASE = 2;
+ static const int32_t MAX_BASE = 36;
+
+private:
+ static const char* _s_alphanumeric_chars;
};
} // namespace doris
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 01c69eb..aa302ce 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -108,6 +108,7 @@ set(VEC_FILES
functions/functions_logical.cpp
functions/function_case.cpp
functions/function_cast.cpp
+ functions/function_conv.cpp
functions/function_string.cpp
functions/function_timestamp.cpp
functions/function_utility.cpp
diff --git a/be/src/vec/data_types/data_type_bitmap.h b/be/src/vec/data_types/data_type_bitmap.h
index 692d6fc..69f5540 100644
--- a/be/src/vec/data_types/data_type_bitmap.h
+++ b/be/src/vec/data_types/data_type_bitmap.h
@@ -18,6 +18,7 @@
#pragma once
#include "util/bitmap_value.h"
#include "vec/columns/column.h"
+#include "vec/columns/column_complex.h"
#include "vec/core/types.h"
#include "vec/data_types/data_type.h"
@@ -27,6 +28,7 @@ public:
DataTypeBitMap() = default;
~DataTypeBitMap() override = default;
+ using ColumnType = ColumnBitmap;
using FieldType = BitmapValue;
std::string do_get_name() const override { return get_family_name(); }
diff --git a/be/src/vec/data_types/data_type_date.h b/be/src/vec/data_types/data_type_date.h
index b3aa90c..b5d148b 100644
--- a/be/src/vec/data_types/data_type_date.h
+++ b/be/src/vec/data_types/data_type_date.h
@@ -34,7 +34,7 @@ public:
bool equals(const IDataType& rhs) const override;
std::string to_string(const IColumn& column, size_t row_num) const;
- void to_string(const IColumn &column, size_t row_num, BufferWritable &ostr) const override;
+ void to_string(const IColumn& column, size_t row_num, BufferWritable& ostr) const override;
static void cast_to_date(Int64& x);
};
diff --git a/be/src/vec/data_types/data_type_date_time.h b/be/src/vec/data_types/data_type_date_time.h
index e2c6bc5..6b6af04 100644
--- a/be/src/vec/data_types/data_type_date_time.h
+++ b/be/src/vec/data_types/data_type_date_time.h
@@ -28,25 +28,25 @@ class DateLUTImpl;
namespace doris::vectorized {
/** DateTime stores time as unix timestamp.
- * The value itself is independent of time zone.
- *
- * In binary format it is represented as unix timestamp.
- * In text format it is serialized to and parsed from YYYY-MM-DD hh:mm:ss format.
- * The text format is dependent of time zone.
- *
- * To convert from/to text format, time zone may be specified explicitly or implicit time zone may be used.
- *
- * Time zone may be specified explicitly as type parameter, example: DateTime('Europe/Moscow').
- * As it does not affect the internal representation of values,
- * all types with different time zones are equivalent and may be used interchangingly.
- * Time zone only affects parsing and displaying in text formats.
- *
- * If time zone is not specified (example: DateTime without parameter), then default time zone is used.
- * Default time zone is server time zone, if server is doing transformations
- * and if client is doing transformations, unless 'use_client_time_zone' setting is passed to client;
- * Server time zone is the time zone specified in 'timezone' parameter in configuration file,
- * or system time zone at the moment of server startup.
- */
+ * The value itself is independent of time zone.
+ *
+ * In binary format it is represented as unix timestamp.
+ * In text format it is serialized to and parsed from YYYY-MM-DD hh:mm:ss format.
+ * The text format is dependent of time zone.
+ *
+ * To convert from/to text format, time zone may be specified explicitly or implicit time zone may be used.
+ *
+ * Time zone may be specified explicitly as type parameter, example: DateTime('Europe/Moscow').
+ * As it does not affect the internal representation of values,
+ * all types with different time zones are equivalent and may be used interchangingly.
+ * Time zone only affects parsing and displaying in text formats.
+ *
+ * If time zone is not specified (example: DateTime without parameter), then default time zone is used.
+ * Default time zone is server time zone, if server is doing transformations
+ * and if client is doing transformations, unless 'use_client_time_zone' setting is passed to client;
+ * Server time zone is the time zone specified in 'timezone' parameter in configuration file,
+ * or system time zone at the moment of server startup.
+ */
class DataTypeDateTime final : public DataTypeNumberBase<Int64> {
public:
DataTypeDateTime();
@@ -62,7 +62,7 @@ public:
std::string to_string(const IColumn& column, size_t row_num) const;
- void to_string(const IColumn &column, size_t row_num, BufferWritable &ostr) const override;
+ void to_string(const IColumn& column, size_t row_num, BufferWritable& ostr) const override;
static void cast_to_date_time(Int64& x);
};
diff --git a/be/src/vec/data_types/data_type_decimal.h b/be/src/vec/data_types/data_type_decimal.h
index 3404eb3..5b6ab86 100644
--- a/be/src/vec/data_types/data_type_decimal.h
+++ b/be/src/vec/data_types/data_type_decimal.h
@@ -88,8 +88,8 @@ class DataTypeDecimal final : public IDataType {
static_assert(IsDecimalNumber<T>);
public:
- using FieldType = T;
using ColumnType = ColumnDecimal<T>;
+ using FieldType = T;
static constexpr bool is_parametric = true;
diff --git a/be/src/vec/data_types/data_type_number_base.h b/be/src/vec/data_types/data_type_number_base.h
index df7d236..e19383e 100644
--- a/be/src/vec/data_types/data_type_number_base.h
+++ b/be/src/vec/data_types/data_type_number_base.h
@@ -35,6 +35,7 @@ class DataTypeNumberBase : public IDataType {
public:
static constexpr bool is_parametric = false;
+ using ColumnType = ColumnVector<T>;
using FieldType = T;
const char* get_family_name() const override { return TypeName<T>::get(); }
@@ -53,7 +54,9 @@ public:
bool is_value_represented_by_number() const override { return true; }
bool is_value_represented_by_integer() const override;
bool is_value_represented_by_unsigned_integer() const override;
- bool is_value_unambiguously_represented_in_contiguous_memory_region() const override { return true; }
+ bool is_value_unambiguously_represented_in_contiguous_memory_region() const override {
+ return true;
+ }
bool have_maximum_size_of_value() const override { return true; }
size_t get_size_of_value_in_memory() const override { return sizeof(T); }
bool is_categorial() const override { return is_value_represented_by_integer(); }
diff --git a/be/src/vec/data_types/data_type_string.h b/be/src/vec/data_types/data_type_string.h
index 0c4ef1e..9d5b21b 100644
--- a/be/src/vec/data_types/data_type_string.h
+++ b/be/src/vec/data_types/data_type_string.h
@@ -28,6 +28,7 @@ namespace doris::vectorized {
class DataTypeString final : public IDataType {
public:
+ using ColumnType = ColumnString;
using FieldType = String;
static constexpr bool is_parametric = false;
@@ -47,7 +48,9 @@ public:
bool have_subtypes() const override { return false; }
bool is_comparable() const override { return true; }
bool can_be_compared_with_collation() const override { return true; }
- bool is_value_unambiguously_represented_in_contiguous_memory_region() const override { return true; }
+ bool is_value_unambiguously_represented_in_contiguous_memory_region() const override {
+ return true;
+ }
bool is_categorial() const override { return true; }
bool can_be_inside_nullable() const override { return true; }
bool can_be_inside_low_cardinality() const override { return true; }
diff --git a/be/src/vec/functions/function_conv.cpp b/be/src/vec/functions/function_conv.cpp
new file mode 100644
index 0000000..6f755ff
--- /dev/null
+++ b/be/src/vec/functions/function_conv.cpp
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/math_functions.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/functions/simple_function_factory.h"
+#include "vec/utils/util.hpp"
+
+namespace doris::vectorized {
+
+template <typename Impl>
+class FunctionConv : public IFunction {
+public:
+ static constexpr auto name = "conv";
+ String get_name() const override { return name; }
+ static FunctionPtr create() { return std::make_shared<FunctionConv<Impl>>(); }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+ DataTypes get_variadic_argument_types_impl() const override {
+ return {std::make_shared<typename Impl::DataType>(), std::make_shared<DataTypeInt8>(),
+ std::make_shared<DataTypeInt8>()};
+ }
+ size_t get_number_of_arguments() const override {
+ return get_variadic_argument_types_impl().size();
+ }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+ bool use_default_implementation_for_constants() const override { return true; }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ auto result_column = ColumnString::create();
+ auto result_null_map_column = ColumnUInt8::create(input_rows_count, 0);
+
+ ColumnPtr argument_columns[3];
+
+ for (int i = 0; i < 3; ++i) {
+ argument_columns[i] = block.get_by_position(arguments[i]).column;
+ if (auto* nullable = check_and_get_column<ColumnNullable>(*argument_columns[i])) {
+ argument_columns[i] = nullable->get_nested_column_ptr();
+ VectorizedUtils::update_null_map(result_null_map_column->get_data(),
+ nullable->get_null_map_data());
+ }
+ }
+
+ execute_straight(
+ context,
+ assert_cast<const typename Impl::DataType::ColumnType*>(argument_columns[0].get()),
+ assert_cast<const ColumnInt8*>(argument_columns[1].get()),
+ assert_cast<const ColumnInt8*>(argument_columns[2].get()),
+ assert_cast<ColumnString*>(result_column.get()),
+ assert_cast<ColumnUInt8*>(result_null_map_column.get())->get_data(),
+ input_rows_count);
+
+ block.get_by_position(result).column =
+ ColumnNullable::create(std::move(result_column), std::move(result_null_map_column));
+ return Status::OK();
+ }
+
+private:
+ void execute_straight(FunctionContext* context,
+ const typename Impl::DataType::ColumnType* data_column,
+ const ColumnInt8* src_base_column, const ColumnInt8* dst_base_column,
+ ColumnString* result_column, NullMap& result_null_map,
+ size_t input_rows_count) {
+ for (size_t i = 0; i < input_rows_count; i++) {
+ if (result_null_map[i]) {
+ result_column->insert_default();
+ continue;
+ }
+
+ Int8 src_base = src_base_column->get_element(i);
+ Int8 dst_base = dst_base_column->get_element(i);
+ if (std::abs(src_base) < MathFunctions::MIN_BASE ||
+ std::abs(src_base) > MathFunctions::MAX_BASE ||
+ std::abs(dst_base) < MathFunctions::MIN_BASE ||
+ std::abs(dst_base) > MathFunctions::MAX_BASE) {
+ result_null_map[i] = true;
+ result_column->insert_default();
+ continue;
+ }
+
+ Impl::calculate_cell(context, data_column, src_base, dst_base, result_column,
+ result_null_map, i);
+ }
+ }
+};
+
+struct ConvInt64Impl {
+ using DataType = DataTypeInt64;
+
+ static void calculate_cell(FunctionContext* context, const DataType::ColumnType* data_column,
+ const Int8 src_base, const Int8 dst_base,
+ ColumnString* result_column, NullMap& result_null_map,
+ size_t index) {
+ Int64 num = data_column->get_element(index);
+ if (src_base < 0 && num >= 0) {
+ result_null_map[index] = true;
+ result_column->insert_default();
+ return;
+ }
+
+ int64_t decimal_num = num;
+ if (src_base != 10) {
+ if (!MathFunctions::decimal_in_base_to_decimal(num, src_base, &decimal_num)) {
+ MathFunctions::handle_parse_result(dst_base, &decimal_num,
+ StringParser::PARSE_OVERFLOW);
+ }
+ }
+ StringVal str = MathFunctions::decimal_to_base(context, decimal_num, dst_base);
+ result_column->insert_data(reinterpret_cast<const char*>(str.ptr), str.len);
+ }
+};
+
+struct ConvStringImpl {
+ using DataType = DataTypeString;
+
+ static void calculate_cell(FunctionContext* context, const DataType::ColumnType* data_column,
+ const Int8 src_base, const Int8 dst_base,
+ ColumnString* result_column, NullMap& result_null_map,
+ size_t index) {
+ StringRef str = data_column->get_data_at(index);
+ StringParser::ParseResult parse_res;
+ int64_t decimal_num =
+ StringParser::string_to_int<int64_t>(str.data, str.size, src_base, &parse_res);
+ if (src_base < 0 && decimal_num >= 0) {
+ result_null_map[index] = true;
+ result_column->insert_default();
+ return;
+ }
+
+ if (!MathFunctions::handle_parse_result(dst_base, &decimal_num, parse_res)) {
+ result_column->insert_data("0", 1);
+ } else {
+ StringVal str = MathFunctions::decimal_to_base(context, decimal_num, dst_base);
+ result_column->insert_data(reinterpret_cast<const char*>(str.ptr), str.len);
+ }
+ }
+};
+
+void register_function_conv(SimpleFunctionFactory& factory) {
+ factory.register_function<FunctionConv<ConvInt64Impl>>();
+ factory.register_function<FunctionConv<ConvStringImpl>>();
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index d757920..0718930 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -36,6 +36,7 @@ void register_function_hll_hash(SimpleFunctionFactory& factory);
void register_function_logical(SimpleFunctionFactory& factory);
void register_function_case(SimpleFunctionFactory& factory);
void register_function_cast(SimpleFunctionFactory& factory);
+void register_function_conv(SimpleFunctionFactory& factory);
void register_function_plus(SimpleFunctionFactory& factory);
void register_function_minus(SimpleFunctionFactory& factory);
void register_function_multiply(SimpleFunctionFactory& factory);
@@ -150,6 +151,7 @@ public:
register_function_logical(instance);
register_function_case(instance);
register_function_cast(instance);
+ register_function_conv(instance);
register_function_plus(instance);
register_function_minus(instance);
register_function_math(instance);
diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index 4adaa79..022b26d 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -452,6 +452,11 @@ module.exports = [
children: ["murmur_hash3_32"],
},
{
+ title: "Math Functions",
+ directoryPath: "math-functions/",
+ children: ["conv"],
+ },
+ {
title: "table functions",
directoryPath: "table-functions/",
children: [
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index fdd2e01..7ec172a 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -444,6 +444,11 @@ module.exports = [
children: ["murmur_hash3_32"],
},
{
+ title: "数学函数",
+ directoryPath: "math-functions/",
+ children: ["conv"],
+ },
+ {
title: "加密和信息摘要函数",
directoryPath: "encrypt-digest-functions/",
children: [
diff --git a/docs/en/sql-reference/sql-functions/math-functions/conv.md b/docs/en/sql-reference/sql-functions/math-functions/conv.md
new file mode 100644
index 0000000..4ce1e18
--- /dev/null
+++ b/docs/en/sql-reference/sql-functions/math-functions/conv.md
@@ -0,0 +1,60 @@
+---
+{
+ "title": "conv",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# conv
+
+## description
+### Syntax
+
+`VARCHAR CONV(VARCHAR input, TINYINT from_base, TINYINT to_base)`
+`VARCHAR CONV(BIGINT input, TINYINT from_base, TINYINT to_base)`
+Convert the input number to the target base. The input base range should be within `[2,36]`.
+
+## example
+
+```
+MySQL [test]> SELECT CONV(15,10,2);
++-----------------+
+| conv(15, 10, 2) |
++-----------------+
+| 1111 |
++-----------------+
+
+MySQL [test]> SELECT CONV('ff',16,10);
++--------------------+
+| conv('ff', 16, 10) |
++--------------------+
+| 255 |
++--------------------+
+
+MySQL [test]> SELECT CONV(230,10,16);
++-------------------+
+| conv(230, 10, 16) |
++-------------------+
+| E6 |
++-------------------+
+```
+
+## keyword
+ CONV
diff --git a/docs/zh-CN/sql-reference/sql-functions/math-functions/conv.md b/docs/zh-CN/sql-reference/sql-functions/math-functions/conv.md
new file mode 100644
index 0000000..280228e
--- /dev/null
+++ b/docs/zh-CN/sql-reference/sql-functions/math-functions/conv.md
@@ -0,0 +1,60 @@
+---
+{
+ "title": "conv",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# conv
+
+## description
+### Syntax
+
+`VARCHAR CONV(VARCHAR input, TINYINT from_base, TINYINT to_base)`
+`VARCHAR CONV(BIGINT input, TINYINT from_base, TINYINT to_base)`
+对输入的数字进行进制转换，输入的进制范围应该在`[2,36]`以内。
+
+## example
+
+```
+MySQL [test]> SELECT CONV(15,10,2);
++-----------------+
+| conv(15, 10, 2) |
++-----------------+
+| 1111 |
++-----------------+
+
+MySQL [test]> SELECT CONV('ff',16,10);
++--------------------+
+| conv('ff', 16, 10) |
++--------------------+
+| 255 |
++--------------------+
+
+MySQL [test]> SELECT CONV(230,10,16);
++-------------------+
+| conv(230, 10, 16) |
++-------------------+
+| E6 |
++-------------------+
+```
+
+## keyword
+ CONV
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 2fe936a..8d3b6d9 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -627,13 +627,13 @@ visible_functions = [
[['conv'], 'VARCHAR', ['BIGINT', 'TINYINT', 'TINYINT'],
'_ZN5doris13MathFunctions8conv_intEPN9doris_udf'
- '15FunctionContextERKNS1_9BigIntValERKNS1_10TinyIntValES9_', '', '', '', ''],
+ '15FunctionContextERKNS1_9BigIntValERKNS1_10TinyIntValES9_', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['conv'], 'VARCHAR', ['VARCHAR', 'TINYINT', 'TINYINT'],
'_ZN5doris13MathFunctions11conv_stringEPN9doris_udf'
- '15FunctionContextERKNS1_9StringValERKNS1_10TinyIntValES9_', '', '', '', ''],
+ '15FunctionContextERKNS1_9StringValERKNS1_10TinyIntValES9_', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['conv'], 'VARCHAR', ['STRING', 'TINYINT', 'TINYINT'],
'_ZN5doris13MathFunctions11conv_stringEPN9doris_udf'
- '15FunctionContextERKNS1_9StringValERKNS1_10TinyIntValES9_', '', '', '', ''],
+ '15FunctionContextERKNS1_9StringValERKNS1_10TinyIntValES9_', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['pmod'], 'BIGINT', ['BIGINT', 'BIGINT'],
'_ZN5doris13MathFunctions11pmod_bigintEPN9doris_udf'
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 04/33: [Bug] Fix bug of concat function and fold const expr (#7608)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5f52c04d62be933c6b1f377db862989a19574778
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Tue Jan 4 06:45:32 2022 -0600
[Bug] Fix bug of concat function and fold const expr (#7608)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/exec/exec_node.cpp | 2 +-
be/src/runtime/fold_constant_executor.cpp | 10 +++++++---
be/src/runtime/fold_constant_executor.h | 2 +-
be/src/vec/exec/join/vhash_join_node.cpp | 9 ++++-----
be/src/vec/exec/vcross_join_node.cpp | 1 -
be/src/vec/functions/function_string.h | 20 ++++----------------
6 files changed, 17 insertions(+), 27 deletions(-)
diff --git a/be/src/exec/exec_node.cpp b/be/src/exec/exec_node.cpp
index b18160e..4582f89 100644
--- a/be/src/exec/exec_node.cpp
+++ b/be/src/exec/exec_node.cpp
@@ -756,7 +756,7 @@ void ExecNode::reached_limit(vectorized::Block* block, bool* eos) {
}
_num_rows_returned += block->rows();
- if (*eos) COUNTER_SET(_rows_returned_counter, _num_rows_returned);
+ COUNTER_SET(_rows_returned_counter, _num_rows_returned);
}
/*
diff --git a/be/src/runtime/fold_constant_executor.cpp b/be/src/runtime/fold_constant_executor.cpp
index cd6d5ff..9781c2f 100644
--- a/be/src/runtime/fold_constant_executor.cpp
+++ b/be/src/runtime/fold_constant_executor.cpp
@@ -82,7 +82,7 @@ Status FoldConstantExecutor::fold_constant_expr(
expr_result.set_success(false);
} else {
expr_result.set_success(true);
- result = _get_result(src, ctx->root()->type().type);
+ result = _get_result(src, 0, ctx->root()->type().type);
}
expr_result.set_content(std::move(result));
@@ -143,7 +143,8 @@ Status FoldConstantExecutor::fold_constant_vexpr(
expr_result.set_success(false);
} else {
expr_result.set_success(true);
- result = _get_result<true>((void *) column_ptr->get_data_at(0).data, ctx->root()->type().type);
+ auto string_ref = column_ptr->get_data_at(0);
+ result = _get_result<true>((void*)string_ref.data, string_ref.size, ctx->root()->type().type);
}
expr_result.set_content(std::move(result));
@@ -198,7 +199,7 @@ Status FoldConstantExecutor::_prepare_and_open(Context* ctx) {
}
template <bool is_vec>
-string FoldConstantExecutor::_get_result(void* src, PrimitiveType slot_type){
+string FoldConstantExecutor::_get_result(void* src, size_t size, PrimitiveType slot_type){
switch (slot_type) {
case TYPE_BOOLEAN: {
bool val = *reinterpret_cast<const bool*>(src);
@@ -237,6 +238,9 @@ string FoldConstantExecutor::_get_result(void* src, PrimitiveType slot_type){
case TYPE_STRING:
case TYPE_HLL:
case TYPE_OBJECT: {
+ if constexpr (is_vec) {
+ return std::string((char*)src, size);
+ }
return (reinterpret_cast<StringValue*>(src))->to_string();
}
case TYPE_DATE:
diff --git a/be/src/runtime/fold_constant_executor.h b/be/src/runtime/fold_constant_executor.h
index c7c5a38..84c52f7 100644
--- a/be/src/runtime/fold_constant_executor.h
+++ b/be/src/runtime/fold_constant_executor.h
@@ -47,7 +47,7 @@ private:
Status _prepare_and_open(Context* ctx);
template <bool is_vec = false>
- std::string _get_result(void* src, PrimitiveType slot_type);
+ std::string _get_result(void* src, size_t size, PrimitiveType slot_type);
std::unique_ptr<RuntimeState> _runtime_state;
std::shared_ptr<MemTracker> _mem_tracker;
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp b/be/src/vec/exec/join/vhash_join_node.cpp
index 62dfffa..7606783 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -124,7 +124,7 @@ struct ProcessRuntimeFilterBuild {
ProcessRuntimeFilterBuild(HashJoinNode* join_node) : _join_node(join_node) {}
Status operator()(RuntimeState* state, HashTableContext& hash_table_ctx) {
- if (_join_node->_runtime_filter_descs.empty() || _join_node->_inserted_rows.empty()) {
+ if (_join_node->_runtime_filter_descs.empty()) {
return Status::OK();
}
VRuntimeFilterSlots* runtime_filter_slots =
@@ -162,7 +162,6 @@ struct ProcessHashTableProbe {
_probe_block(join_node->_probe_block),
_probe_index(join_node->_probe_index),
_probe_raw_ptrs(join_node->_probe_columns),
- _arena(join_node->_arena),
_rows_returned_counter(join_node->_rows_returned_counter) {}
// Only process the join with no other join conjunt, because of no other join conjunt
@@ -198,7 +197,7 @@ struct ProcessHashTableProbe {
_arena)) {nullptr, false}
: key_getter.find_key(hash_table_ctx.hash_table, _probe_index, _arena);
- if (_probe_index + 4 < _probe_rows)
+ if (_probe_index + 2 < _probe_rows)
key_getter.prefetch(hash_table_ctx.hash_table, _probe_index + 2, _arena);
if (find_result.is_found()) {
@@ -218,7 +217,7 @@ struct ProcessHashTableProbe {
if (!_join_node->_is_right_semi_anti) {
++current_offset;
for (size_t j = 0; j < right_col_len; ++j) {
- auto &column = *mapped.block->get_by_position(j).column;
+ auto& column = *mapped.block->get_by_position(j).column;
mcol[j + right_col_idx]->insert_from(column, mapped.row_num);
}
}
@@ -515,7 +514,7 @@ private:
const Block& _probe_block;
int& _probe_index;
ColumnRawPtrs& _probe_raw_ptrs;
- Arena& _arena;
+ Arena _arena;
ProfileCounter* _rows_returned_counter;
};
diff --git a/be/src/vec/exec/vcross_join_node.cpp b/be/src/vec/exec/vcross_join_node.cpp
index 7a5086e..6d48527 100644
--- a/be/src/vec/exec/vcross_join_node.cpp
+++ b/be/src/vec/exec/vcross_join_node.cpp
@@ -135,7 +135,6 @@ Status VCrossJoinNode::get_next(RuntimeState* state, Block* block, bool* eos) {
}
}
dst_columns.clear();
-
RETURN_IF_ERROR(VExprContext::filter_block(_vconjunct_ctx_ptr, block, block->columns()));
reached_limit(block, eos);
diff --git a/be/src/vec/functions/function_string.h b/be/src/vec/functions/function_string.h
index 2893fe1..efeef41 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -326,9 +326,9 @@ public:
bool is_variadic() const override { return true; }
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
- return make_nullable(std::make_shared<DataTypeString>());
+ return std::make_shared<DataTypeString>();
}
- bool use_default_implementation_for_nulls() const override { return false; }
+ bool use_default_implementation_for_nulls() const override { return true; }
bool use_default_implementation_for_constants() const override { return true; }
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
@@ -336,16 +336,10 @@ public:
DCHECK_GE(arguments.size(), 1);
if (arguments.size() == 1) {
- if (block.get_by_position(arguments[0]).column->is_nullable()) {
- block.get_by_position(result).column = block.get_by_position(arguments[0]).column;
- } else {
- block.get_by_position(result).column =
- make_nullable(block.get_by_position(arguments[0]).column);
- }
+ block.get_by_position(result).column = block.get_by_position(arguments[0]).column;
return Status::OK();
}
- auto null_map = ColumnUInt8::create(input_rows_count, 0);
int argument_size = arguments.size();
ColumnPtr argument_columns[argument_size];
@@ -355,11 +349,6 @@ public:
for (int i = 0; i < argument_size; ++i) {
argument_columns[i] =
block.get_by_position(arguments[i]).column->convert_to_full_column_if_const();
- if (auto* nullable = check_and_get_column<const ColumnNullable>(*argument_columns[i])) {
- argument_columns[i] = nullable->get_nested_column_ptr();
- VectorizedUtils::update_null_map(null_map->get_data(),
- nullable->get_null_map_data());
- }
auto col_str = assert_cast<const ColumnString*>(argument_columns[i].get());
offsets_list[i] = &col_str->get_offsets();
chars_list[i] = &col_str->get_chars();
@@ -401,8 +390,7 @@ public:
res_offset[i] = res_offset[i - 1] + current_length;
}
- block.get_by_position(result).column =
- ColumnNullable::create(std::move(res), std::move(null_map));
+ block.get_by_position(result).column = std::move(res);
return Status::OK();
}
};
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 06/33: [Vectorized][Function] Support function and (#7618)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5779e9b580d8c4b80e9f3f677974d1b05ee4215a
Author: Pxl <95...@qq.com>
AuthorDate: Wed Jan 5 20:11:07 2022 +0800
[Vectorized][Function] Support function and (#7618)
---
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/functions/function_utility.cpp | 118 +++++++++++++++++++++++++
be/src/vec/functions/math.cpp | 1 +
be/src/vec/functions/simple_function_factory.h | 2 +
gensrc/script/doris_builtins_functions.py | 4 +-
5 files changed, 124 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 1738819..71efde5 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -110,6 +110,7 @@ set(VEC_FILES
functions/function_cast.cpp
functions/function_string.cpp
functions/function_timestamp.cpp
+ functions/function_utility.cpp
functions/comparison_equal_for_null.cpp
functions/function_json.cpp
functions/hll_cardinality.cpp
diff --git a/be/src/vec/functions/function_utility.cpp b/be/src/vec/functions/function_utility.cpp
new file mode 100644
index 0000000..6c7da89
--- /dev/null
+++ b/be/src/vec/functions/function_utility.cpp
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "util/monotime.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/functions/simple_function_factory.h"
+
+namespace doris::vectorized {
+
+class FunctionSleep : public IFunction {
+public:
+ static constexpr auto name = "sleep";
+ static FunctionPtr create() { return std::make_shared<FunctionSleep>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 1; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ if (arguments[0].get()->is_nullable()) {
+ return make_nullable(std::make_shared<DataTypeUInt8>());
+ }
+ return std::make_shared<DataTypeUInt8>();
+ }
+
+ bool use_default_implementation_for_constants() const override { return true; }
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ ColumnPtr argument_column =
+ block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();
+
+ auto res_column = ColumnUInt8::create();
+
+ if (auto* nullable_column = check_and_get_column<ColumnNullable>(*argument_column)) {
+ auto null_map_column = ColumnUInt8::create();
+
+ auto nested_column = nullable_column->get_nested_column_ptr();
+ auto data_column = assert_cast<const ColumnVector<Int32>*>(nested_column.get());
+
+ for (int i = 0; i < input_rows_count; i++) {
+ if (nullable_column->is_null_at(i)) {
+ res_column->insert(0);
+ null_map_column->insert(1);
+ } else {
+ int seconds = data_column->get_data()[i];
+ SleepFor(MonoDelta::FromSeconds(seconds));
+ res_column->insert(1);
+ null_map_column->insert(0);
+ }
+ }
+
+ block.replace_by_position(result, ColumnNullable::create(std::move(res_column),
+ std::move(null_map_column)));
+ } else {
+ auto data_column = assert_cast<const ColumnVector<Int32>*>(argument_column.get());
+
+ for (int i = 0; i < input_rows_count; i++) {
+ int seconds = data_column->get_element(i);
+ SleepFor(MonoDelta::FromSeconds(seconds));
+ res_column->insert(1);
+ }
+
+ block.replace_by_position(result, std::move(res_column));
+ }
+ return Status::OK();
+ }
+};
+
+class FunctionVersion : public IFunction {
+public:
+ static constexpr auto name = "version";
+
+ static const std::string version;
+
+ static FunctionPtr create() { return std::make_shared<FunctionVersion>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 0; }
+
+ DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments) const override {
+ return std::make_shared<DataTypeString>();
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ auto res_column = ColumnString::create();
+ res_column->insert_data(version.c_str(), version.length());
+ block.replace_by_position(result, std::move(res_column));
+ return Status::OK();
+ }
+};
+
+const std::string FunctionVersion::version = "5.1.0";
+
+void register_function_utility(SimpleFunctionFactory& factory) {
+ factory.register_function<FunctionSleep>();
+ factory.register_function<FunctionVersion>();
+}
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/functions/math.cpp b/be/src/vec/functions/math.cpp
index 57d6c48..4aa156a 100644
--- a/be/src/vec/functions/math.cpp
+++ b/be/src/vec/functions/math.cpp
@@ -455,6 +455,7 @@ void register_function_math(SimpleFunctionFactory& factory) {
factory.register_function<FunctionPositive>();
factory.register_function<FunctionSin>();
factory.register_function<FunctionSqrt>();
+ factory.register_alias("sqrt", "dsqrt");
factory.register_function<FunctionTan>();
factory.register_function<FunctionFloor>();
factory.register_alias("floor", "dfloor");
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index 716a644..149ce88 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -57,6 +57,7 @@ void register_function_if(SimpleFunctionFactory& factory);
void register_function_nullif(SimpleFunctionFactory& factory);
void register_function_date_time_computation(SimpleFunctionFactory& factory);
void register_function_timestamp(SimpleFunctionFactory& factory);
+void register_function_utility(SimpleFunctionFactory& factory);
void register_function_json(SimpleFunctionFactory& factory);
void register_function_function_hash(SimpleFunctionFactory& factory);
void register_function_function_ifnull(SimpleFunctionFactory& factory);
@@ -159,6 +160,7 @@ public:
register_function_nullif(instance);
register_function_date_time_computation(instance);
register_function_timestamp(instance);
+ register_function_utility(instance);
register_function_date_time_to_string(instance);
register_function_date_time_string_to_string(instance);
register_function_json(instance);
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 8ac3b15..7bf7d2d 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -1072,10 +1072,10 @@ visible_functions = [
# Utility functions
[['sleep'], 'BOOLEAN', ['INT'],
'_ZN5doris16UtilityFunctions5sleepEPN9doris_udf15FunctionContextERKNS1_6IntValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['version'], 'VARCHAR', [],
'_ZN5doris16UtilityFunctions7versionEPN9doris_udf15FunctionContextE',
- '', '', '', 'ALWAYS_NOT_NULLABLE'],
+ '', '', 'vec', 'ALWAYS_NOT_NULLABLE'],
# Json functions
[['get_json_int'], 'INT', ['VARCHAR', 'VARCHAR'],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 27/33: [Vectorization] Support SegmentIterator vectorization (#7613)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit e23e332f75f6d1b3891c9adcd7ba85f9a89f765e
Author: wangbo <50...@qq.com>
AuthorDate: Fri Jan 14 11:44:27 2022 +0800
[Vectorization] Support SegmentIterator vectorization (#7613)
---
be/src/olap/column_predicate.h | 2 +
be/src/olap/comparison_predicate.cpp | 2 +-
be/src/olap/in_list_predicate.cpp | 39 +++
be/src/olap/in_list_predicate.h | 7 +-
be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 21 +-
be/src/olap/rowset/segment_v2/binary_plain_page.h | 34 +-
be/src/olap/rowset/segment_v2/bitshuffle_page.h | 36 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 362 ++++++++++++++++++++-
be/src/olap/rowset/segment_v2/segment_iterator.h | 29 +-
be/src/olap/schema.cpp | 66 ++++
be/src/olap/schema.h | 4 +
be/src/vec/columns/column.h | 13 +-
be/src/vec/columns/column_complex.h | 2 +
be/src/vec/columns/column_nullable.cpp | 15 +-
be/src/vec/columns/column_nullable.h | 6 +
be/src/vec/columns/column_vector.h | 26 ++
be/src/vec/columns/predicate_column.h | 27 +-
17 files changed, 661 insertions(+), 30 deletions(-)
diff --git a/be/src/olap/column_predicate.h b/be/src/olap/column_predicate.h
index 10b8a91..6b1aa23 100644
--- a/be/src/olap/column_predicate.h
+++ b/be/src/olap/column_predicate.h
@@ -69,6 +69,8 @@ public:
virtual void evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const {};
uint32_t column_id() const { return _column_id; }
+ virtual bool is_in_predicate() { return false; }
+
protected:
uint32_t _column_id;
bool _opposite;
diff --git a/be/src/olap/comparison_predicate.cpp b/be/src/olap/comparison_predicate.cpp
index 598e7f3..a154a04 100644
--- a/be/src/olap/comparison_predicate.cpp
+++ b/be/src/olap/comparison_predicate.cpp
@@ -188,7 +188,7 @@ COMPARISON_PRED_COLUMN_EVALUATE(GreaterEqualPredicate, >=)
void CLASS<type>::evaluate_vec(vectorized::IColumn& column, uint16_t size, bool* flags) const { \
if (column.is_nullable()) { \
auto* nullable_column = vectorized::check_and_get_column<vectorized::ColumnNullable>(column); \
- auto& data_array = reinterpret_cast<const vectorized::ColumnVector<type>&>(nullable_column->get_nested_column()).get_data(); \
+ auto& data_array = reinterpret_cast<const vectorized::PredicateColumnType<type>&>(nullable_column->get_nested_column()).get_data(); \
auto& null_bitmap = reinterpret_cast<const vectorized::ColumnVector<uint8_t>&>(*(nullable_column->get_null_map_column_ptr())).get_data(); \
for (uint16_t i = 0; i < size; i++) { \
flags[i] = (data_array[i] OP _value) && (!null_bitmap[i]); \
diff --git a/be/src/olap/in_list_predicate.cpp b/be/src/olap/in_list_predicate.cpp
index c167a17..a17e157 100644
--- a/be/src/olap/in_list_predicate.cpp
+++ b/be/src/olap/in_list_predicate.cpp
@@ -20,6 +20,8 @@
#include "olap/field.h"
#include "runtime/string_value.hpp"
#include "runtime/vectorized_row_batch.h"
+#include "vec/columns/predicate_column.h"
+#include "vec/columns/column_nullable.h"
namespace doris {
@@ -115,6 +117,43 @@ IN_LIST_PRED_EVALUATE(NotInListPredicate, ==)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(InListPredicate, !=)
IN_LIST_PRED_COLUMN_BLOCK_EVALUATE(NotInListPredicate, ==)
+#define IN_LIST_PRED_COLUMN_EVALUATE(CLASS, OP) \
+ template <class type> \
+ void CLASS<type>::evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const { \
+ uint16_t new_size = 0; \
+ if (column.is_nullable()) { \
+ auto* nullable_column = \
+ vectorized::check_and_get_column<vectorized::ColumnNullable>(column); \
+ auto& null_bitmap = reinterpret_cast<const vectorized::ColumnVector<uint8_t>&>(*( \
+ nullable_column->get_null_map_column_ptr())).get_data(); \
+ auto* nest_column_vector = vectorized::check_and_get_column \
+ <vectorized::PredicateColumnType<type>>(nullable_column->get_nested_column()); \
+ auto& data_array = nest_column_vector->get_data(); \
+ for (uint16_t i = 0; i < *size; i++) { \
+ uint16_t idx = sel[i]; \
+ sel[new_size] = idx; \
+ const type& cell_value = reinterpret_cast<const type&>(data_array[idx]); \
+ bool ret = !null_bitmap[idx] && (_values.find(cell_value) OP _values.end()); \
+ new_size += _opposite ? !ret : ret; \
+ } \
+ *size = new_size; \
+ } else { \
+ auto& number_column = reinterpret_cast<vectorized::PredicateColumnType<type>&>(column);\
+ auto& data_array = number_column.get_data(); \
+ for (uint16_t i = 0; i < *size; i++) { \
+ uint16_t idx = sel[i]; \
+ sel[new_size] = idx; \
+ const type& cell_value = reinterpret_cast<const type&>(data_array[idx]); \
+ auto result = (_values.find(cell_value) OP _values.end()); \
+ new_size += _opposite ? !result : result; \
+ } \
+ } \
+ *size = new_size; \
+ }
+
+IN_LIST_PRED_COLUMN_EVALUATE(InListPredicate, !=)
+IN_LIST_PRED_COLUMN_EVALUATE(NotInListPredicate, ==)
+
#define IN_LIST_PRED_COLUMN_BLOCK_EVALUATE_OR(CLASS, OP) \
template <class type> \
void CLASS<type>::evaluate_or(ColumnBlock* block, uint16_t* sel, uint16_t size, bool* flags) \
diff --git a/be/src/olap/in_list_predicate.h b/be/src/olap/in_list_predicate.h
index cf9bf61..7cd237b 100644
--- a/be/src/olap/in_list_predicate.h
+++ b/be/src/olap/in_list_predicate.h
@@ -76,6 +76,8 @@ namespace doris {
class VectorizedRowBatch;
+// todo(wb) support evaluate_and,evaluate_or
+
#define IN_LIST_PRED_CLASS_DEFINE(CLASS) \
template <class type> \
class CLASS : public ColumnPredicate { \
@@ -90,7 +92,10 @@ class VectorizedRowBatch;
virtual Status evaluate(const Schema& schema, \
const std::vector<BitmapIndexIterator*>& iterators, \
uint32_t num_rows, roaring::Roaring* bitmap) const override; \
- \
+ void evaluate(vectorized::IColumn& column, uint16_t* sel, uint16_t* size) const override; \
+ void evaluate_and(vectorized::IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const override {} \
+ void evaluate_or(vectorized::IColumn& column, uint16_t* sel, uint16_t size, bool* flags) const override {} \
+ bool is_in_predicate() override { return true; } \
private: \
phmap::flat_hash_set<type> _values; \
};
diff --git a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
index 7825dcc..edd64db 100644
--- a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
+++ b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
@@ -257,15 +257,26 @@ Status BinaryDictPageDecoder::next_batch(size_t* n, vectorized::MutableColumnPtr
const int32_t* data_array = reinterpret_cast<const int32_t*>(_bit_shuffle_ptr->_chunk.data);
size_t start_index = _bit_shuffle_ptr->_cur_index;
- // todo(wb) support nullable
- if (dst->is_predicate_column()) {
+ auto* dst_col_ptr = dst.get();
+ if (dst->is_nullable()) {
+ auto nullable_column = assert_cast<vectorized::ColumnNullable*>(dst.get());
+ dst_col_ptr = nullable_column->get_nested_column_ptr().get();
+
+ // fill null bitmap here, not null;
+ // todo(wb) using SIMD speed up here
+ for (int i = 0; i < max_fetch; i++) {
+ nullable_column->get_null_map_data().push_back(0);
+ }
+ }
+
+ if (dst_col_ptr->is_predicate_column()) {
// cast columnptr to columnstringvalue just for avoid virtual function call overhead
- vectorized::ColumnStringValue& string_value_vector = reinterpret_cast<vectorized::ColumnStringValue&>(*dst);
+ auto* string_value_column_ptr = reinterpret_cast<vectorized::ColumnStringValue*>(dst_col_ptr);
for (int i = 0; i < max_fetch; i++, start_index++) {
int32_t codeword = data_array[start_index];
uint32_t start_offset = _start_offset_array[codeword];
uint32_t str_len = _len_array[codeword];
- string_value_vector.insert_data(&_dict_decoder->_data[start_offset], str_len);
+ string_value_column_ptr->insert_data(&_dict_decoder->_data[start_offset], str_len);
}
} else {
// todo(wb) research whether using batch memcpy to insert columnString can has better performance when data set is big
@@ -273,7 +284,7 @@ Status BinaryDictPageDecoder::next_batch(size_t* n, vectorized::MutableColumnPtr
int32_t codeword = data_array[start_index];
const uint32_t start_offset = _start_offset_array[codeword];
const uint32_t str_len = _len_array[codeword];
- dst->insert_data(&_dict_decoder->_data[start_offset], str_len);
+ dst_col_ptr->insert_data(&_dict_decoder->_data[start_offset], str_len);
}
}
_bit_shuffle_ptr->_cur_index += max_fetch;
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 25ce9e0..3c55d2b 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -38,6 +38,8 @@
#include "runtime/mem_pool.h"
#include "util/coding.h"
#include "util/faststring.h"
+#include "vec/columns/column_complex.h"
+#include "vec/columns/column_nullable.h"
namespace doris {
namespace segment_v2 {
@@ -235,20 +237,46 @@ public:
}
const size_t max_fetch = std::min(*n, static_cast<size_t>(_num_elems - _cur_idx));
- if (dst->is_predicate_column()) {
+ auto* dst_col_ptr = dst.get();
+ if (dst->is_nullable()) {
+ auto nullable_column = assert_cast<vectorized::ColumnNullable*>(dst.get());
+ dst_col_ptr = nullable_column->get_nested_column_ptr().get();
+ // fill null bitmap here, not null;
+ for (int i = 0; i < max_fetch; i++) {
+ nullable_column->get_null_map_data().push_back(0);
+ }
+ }
+
+ if (dst_col_ptr->is_bitmap()) {
+ auto& bitmap_column = reinterpret_cast<vectorized::ColumnBitmap&>(*dst_col_ptr);
+ for (size_t i = 0; i < max_fetch; i++, _cur_idx++) {
+ const uint32_t start_offset = offset(_cur_idx);
+ uint32_t len = offset(_cur_idx + 1) - start_offset;
+
+ bitmap_column.insert_default();
+ BitmapValue* pvalue = &bitmap_column.get_element(bitmap_column.size() - 1);
+ if (len != 0) {
+ BitmapValue value;
+ value.deserialize(&_data[start_offset]);
+ *pvalue = std::move(value);
+ } else {
+ *pvalue = std::move(*reinterpret_cast<BitmapValue*>(const_cast<char*>(&_data[start_offset])));
+ }
+ }
+ } else if (dst_col_ptr->is_predicate_column()) {
// todo(wb) padding sv here for better comparison performance
for (size_t i = 0; i < max_fetch; i++, _cur_idx++) {
const uint32_t start_offset = offset(_cur_idx);
uint32_t len = offset(_cur_idx + 1) - start_offset;
StringValue sv(const_cast<char*>(&_data[start_offset]), len);
- dst->insert_data(reinterpret_cast<char*>(&sv), 0);
+ dst_col_ptr->insert_data(reinterpret_cast<char*>(&sv), 0);
}
} else {
for (size_t i = 0; i < max_fetch; i++, _cur_idx++) {
// todo(wb) need more test case and then improve here
const uint32_t start_offset = offset(_cur_idx);
uint32_t len = offset(_cur_idx + 1) - start_offset;
- dst->insert_data(&_data[start_offset], len);
+ dst_col_ptr->insert_data(&_data[start_offset], len);
}
}
diff --git a/be/src/olap/rowset/segment_v2/bitshuffle_page.h b/be/src/olap/rowset/segment_v2/bitshuffle_page.h
index 5afbb7d..83eebae 100644
--- a/be/src/olap/rowset/segment_v2/bitshuffle_page.h
+++ b/be/src/olap/rowset/segment_v2/bitshuffle_page.h
@@ -37,6 +37,8 @@
#include "util/coding.h"
#include "util/faststring.h"
#include "util/slice.h"
+#include "vec/runtime/vdatetime_value.h"
+#include "vec/columns/column_nullable.h"
namespace doris {
namespace segment_v2 {
@@ -359,17 +361,28 @@ public:
int begin = _cur_index;
int end = _cur_index + max_fetch;
-
+
+ auto* dst_col_ptr = dst.get();
+ if (dst->is_nullable()) {
+ auto nullable_column = assert_cast<vectorized::ColumnNullable*>(dst.get());
+ dst_col_ptr = nullable_column->get_nested_column_ptr().get();
+
+ // fill null bitmap here, not null;
+ for (int j = begin; j < end; j++) {
+ nullable_column->get_null_map_data().push_back(0);
+ }
+ }
+
// todo(wb) Try to eliminate type judgment in pagedecoder
- if (dst->is_column_decimal()) { // decimal non-predicate column
+ if (dst_col_ptr->is_column_decimal()) { // decimal non-predicate column
for (; begin < end; begin++) {
const char* cur_ptr = (const char*)&_chunk.data[begin * SIZE_OF_TYPE];
int64_t int_value = *(int64_t*)(cur_ptr);
int32_t frac_value = *(int32_t*)(cur_ptr + sizeof(int64_t));
DecimalV2Value data(int_value, frac_value);
- dst->insert_data(reinterpret_cast<char*>(&data), 0);
+ dst_col_ptr->insert_data(reinterpret_cast<char*>(&data), 0);
}
- } else if (dst->is_date_type()) {
+ } else if (dst_col_ptr->is_date_type()) {
for (; begin < end; begin++) {
const char* cur_ptr = (const char*)&_chunk.data[begin * SIZE_OF_TYPE];
uint64_t value = 0;
@@ -378,14 +391,21 @@ public:
value |= *(unsigned char*)(cur_ptr + 1);
value <<= 8;
value |= *(unsigned char*)(cur_ptr);
- DateTimeValue date;
+ vectorized::VecDateTimeValue date;
date.from_olap_date(value);
- dst->insert_data(reinterpret_cast<char*>(&date), 0);
+ dst_col_ptr->insert_data(reinterpret_cast<char*>(&date), 0);
+ }
+ } else if (dst_col_ptr->is_datetime_type()) {
+ for (; begin < end; begin++) {
+ const char* cur_ptr = (const char*)&_chunk.data[begin * SIZE_OF_TYPE];
+ uint64_t value = *reinterpret_cast<const uint64_t*>(cur_ptr);
+ vectorized::VecDateTimeValue date(value);
+ dst_col_ptr->insert_data(reinterpret_cast<char*>(&date), 0);
}
} else {
- // todo(wb) need performance test here, may be batch memory copy?
+ // todo(wb) batch insert here
for (; begin < end; begin++) {
- dst->insert_data((const char*)&_chunk.data[begin * SIZE_OF_TYPE], 0);
+ dst_col_ptr->insert_data((const char*)&_chunk.data[begin * SIZE_OF_TYPE], 0);
}
}
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index d19f56e..86a89ac 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -30,6 +30,7 @@
#include "olap/rowset/segment_v2/segment.h"
#include "olap/short_key_index.h"
#include "util/doris_metrics.h"
+#include "olap/in_list_predicate.h"
using strings::Substitute;
@@ -120,7 +121,7 @@ Status SegmentIterator::init(const StorageReadOptions& opts) {
return Status::OK();
}
-Status SegmentIterator::_init() {
+Status SegmentIterator::_init(bool is_vec) {
DorisMetrics::instance()->segment_read_total->increment(1);
// get file handle from file descriptor of segment
fs::BlockManager* block_mgr = fs::fs_util::block_manager(_segment->_path_desc.storage_medium);
@@ -133,7 +134,11 @@ Status SegmentIterator::_init() {
RETURN_IF_ERROR(_get_row_ranges_by_keys());
}
RETURN_IF_ERROR(_get_row_ranges_by_column_conditions());
- _init_lazy_materialization();
+ if (is_vec) {
+ _vec_init_lazy_materialization();
+ } else {
+ _init_lazy_materialization();
+ }
_range_iter.reset(new BitmapRangeIterator(_row_bitmap));
return Status::OK();
}
@@ -581,9 +586,358 @@ Status SegmentIterator::next_batch(RowBlockV2* block) {
return Status::OK();
}
+/* ---------------------- for vecterization implementation ---------------------- */
+
+// todo(wb) need a UT here
+void SegmentIterator::_vec_init_lazy_materialization() {
+ _is_pred_column.resize(_schema.columns().size(), false);
+
+ std::set<ColumnId> pred_column_ids; // including short_cir_pred_col_id_set and vec_pred_col_id_set
+ _is_all_column_basic_type = true;
+ bool is_predicate_column_exists = false;
+ bool is_non_predicate_column_exists = false;
+
+ if (!_col_predicates.empty()) {
+ is_predicate_column_exists = true;
+
+ std::set<ColumnId> short_cir_pred_col_id_set; // using set for distinct cid
+ std::set<ColumnId> vec_pred_col_id_set;
+
+ for (auto predicate : _col_predicates) {
+ auto cid = predicate->column_id();
+ FieldType type = _schema.column(cid)->type();
+ _is_pred_column[cid] = true;
+ pred_column_ids.insert(cid);
+
+ // for date type which can not be executed in a vectorized way, using short circuit execution
+ if (type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR || type == OLAP_FIELD_TYPE_DECIMAL
+ || type == OLAP_FIELD_TYPE_DATE || predicate->is_in_predicate()) {
+ short_cir_pred_col_id_set.insert(cid);
+ _short_cir_eval_predicate.push_back(predicate);
+ _is_all_column_basic_type = false;
+ } else {
+ vec_pred_col_id_set.insert(predicate->column_id());
+ if (_pre_eval_block_predicate == nullptr) {
+ _pre_eval_block_predicate = new AndBlockColumnPredicate();
+ }
+ reinterpret_cast<MutilColumnBlockPredicate*>(_pre_eval_block_predicate)->add_column_predicate(new SingleColumnBlockPredicate(predicate));
+ }
+ }
+
+ std::set<ColumnId> del_cond_id_set;
+ _opts.delete_condition_predicates.get()->get_all_column_ids(del_cond_id_set);
+ short_cir_pred_col_id_set.insert(del_cond_id_set.begin(), del_cond_id_set.end());
+ pred_column_ids.insert(del_cond_id_set.begin(), del_cond_id_set.end());
+
+ if (_schema.column_ids().size() > pred_column_ids.size()) {
+ for (auto cid : _schema.column_ids()) {
+ if (!_is_pred_column[cid]) {
+ _non_predicate_columns.push_back(cid);
+ is_non_predicate_column_exists = true;
+ }
+ }
+ }
+
+ _vec_pred_column_ids.assign(vec_pred_col_id_set.cbegin(), vec_pred_col_id_set.cend());
+ _short_cir_pred_column_ids.assign(short_cir_pred_col_id_set.cbegin(), short_cir_pred_col_id_set.cend());
+ } else {
+ _is_all_column_basic_type = false;
+ is_non_predicate_column_exists = true;
+ for (auto cid : _schema.column_ids()) {
+ _non_predicate_columns.push_back(cid);
+ }
+ }
+
+ // note(wb) in following cases we disable lazy materialization
+ // case 1: when all column is basic type(is_all_column_basic_type = true)
+ // because we think `seek and read` cost > read page cost, lazy materialize may cause more `seek and read`, so disable it
+ // case 2: all column is predicate column
+ // case 3: all column is not predicate column
+ // todo(wb) need further research more lazy materialization rule, such as get more info from `statistics` for better decision
+ if (_is_all_column_basic_type) {
+ std::set<ColumnId> pred_set(_vec_pred_column_ids.begin(), _vec_pred_column_ids.end());
+ std::set<ColumnId> non_pred_set(_non_predicate_columns.begin(), _non_predicate_columns.end());
+
+ // when _is_all_column_basic_type = true, _first_read_column_ids should keep the same order with _schema.column_ids which stands for return column order
+ for (int i = 0; i < _schema.num_column_ids(); i++) {
+ auto cid = _schema.column_ids()[i];
+ if (pred_set.find(cid) != pred_set.end()) {
+ _first_read_column_ids.push_back(cid);
+ } else if (non_pred_set.find(cid) != non_pred_set.end()) {
+ _first_read_column_ids.push_back(cid);
+ _is_pred_column[cid] = true; // in this case, non-predicate column should also be filtered by sel idx, so we regard it as pred columns
+ }
+ }
+
+ } else if (is_predicate_column_exists && !is_non_predicate_column_exists) {
+ _first_read_column_ids.assign(pred_column_ids.cbegin(), pred_column_ids.cend());
+ } else if (!is_predicate_column_exists && is_non_predicate_column_exists) {
+ for (auto cid : _non_predicate_columns) {
+ _first_read_column_ids.push_back(cid);
+ }
+ } else {
+ _lazy_materialization_read = true;
+ _first_read_column_ids.assign(pred_column_ids.cbegin(), pred_column_ids.cend());
+ }
+
+ // make _schema_block_id_map
+ _schema_block_id_map.resize(_schema.columns().size());
+ for (int i = 0; i < _schema.num_column_ids(); i++) {
+ auto cid = _schema.column_ids()[i];
+ _schema_block_id_map[cid] = i;
+ }
+
+}
+
+Status SegmentIterator::_read_columns(const std::vector<ColumnId>& column_ids, vectorized::MutableColumns& column_block, size_t nrows) {
+ for (auto cid : column_ids) {
+ auto& column = column_block[cid];
+ size_t rows_read = nrows;
+ RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&rows_read, column));
+ DCHECK_EQ(nrows, rows_read);
+ }
+ return Status::OK();
+}
+
+void SegmentIterator::_init_current_block(vectorized::Block* block, std::vector<vectorized::MutableColumnPtr>& current_columns) {
+ bool is_block_mem_reuse= block->mem_reuse();
+ if (is_block_mem_reuse) {
+ size_t column_to_keep = _schema.num_column_ids();
+ for (int i = block->columns() - 1; i >= column_to_keep; i--) {
+ block->erase(i);
+ }
+ block->clear_column_data();
+ } else { // pre fill output block here
+ for (size_t i = 0; i < _schema.num_column_ids(); i++) {
+ auto cid = _schema.column_ids()[i];
+ auto* column_desc = _schema.columns()[cid];
+ auto data_type = Schema::get_data_type_ptr(column_desc->type());
+ if (column_desc->is_nullable()) {
+ block->insert({nullptr, std::make_shared<vectorized::DataTypeNullable>(std::move(data_type)), column_desc->name()});
+ } else {
+ block->insert({nullptr, std::move(data_type), column_desc->name()});
+ }
+ }
+ }
+
+ for (size_t i = 0; i < _schema.num_column_ids(); i++) {
+ auto cid = _schema.column_ids()[i];
+ if (_is_pred_column[cid]) { //todo(wb) maybe we can relase it after output block
+ current_columns[cid]->clear();
+ } else { // non-predicate column
+ auto &column_desc = _schema.columns()[cid];
+ if (is_block_mem_reuse) {
+ current_columns[cid] = std::move(*block->get_by_position(i).column).mutate();
+ } else {
+ auto data_type = Schema::get_data_type_ptr(column_desc->type());
+ if (column_desc->is_nullable()) {
+ current_columns[cid] = doris::vectorized::ColumnNullable::create(
+ std::move(data_type->create_column()), doris::vectorized::ColumnUInt8::create());
+ } else {
+ current_columns[cid] = data_type->create_column();
+ }
+ }
+ if (column_desc->type() == OLAP_FIELD_TYPE_DATE) {
+ current_columns[cid]->set_date_type();
+ } else if (column_desc->type() == OLAP_FIELD_TYPE_DATETIME) {
+ current_columns[cid]->set_datetime_type();
+ }
+ }
+ }
+}
+
+void SegmentIterator::_output_non_pred_columns(vectorized::Block* block, bool is_block_mem_reuse) {
+ for (auto cid : _non_predicate_columns) {
+ block->replace_by_position(_schema_block_id_map[cid], std::move(_current_return_columns[cid]));
+ }
+ }
+
+void SegmentIterator::_output_column_by_sel_idx(vectorized::Block* block, std::vector<ColumnId> columnIds,
+ uint16_t* sel_rowid_idx, uint16_t select_size, bool is_block_mem_reuse) {
+ for (auto cid : columnIds) {
+ auto &column_ptr = _current_return_columns[cid];
+ if (is_block_mem_reuse) {
+ column_ptr->filter_by_selector(sel_rowid_idx, select_size,
+ &block->get_by_position(_schema_block_id_map[cid]).column);
+ } else {
+ block->replace_by_position(_schema_block_id_map[cid],
+ (*column_ptr).get_ptr()->filter_by_selector(sel_rowid_idx, select_size));
+ }
+ }
+ }
+
+
+Status SegmentIterator::_read_columns_by_index(uint32_t nrows_read_limit, uint32_t& nrows_read, bool set_block_rowid) {
+ do {
+ uint32_t range_from;
+ uint32_t range_to;
+ bool has_next_range =
+ _range_iter->next_range(nrows_read_limit - nrows_read, &range_from, &range_to);
+ if (!has_next_range) {
+ break;
+ }
+ if (_cur_rowid == 0 || _cur_rowid != range_from) {
+ _cur_rowid = range_from;
+ RETURN_IF_ERROR(_seek_columns(_first_read_column_ids, _cur_rowid));
+ }
+ size_t rows_to_read = range_to - range_from;
+ RETURN_IF_ERROR(_read_columns(_first_read_column_ids, _current_return_columns, rows_to_read));
+ _cur_rowid += rows_to_read;
+ if (set_block_rowid) {
+ for (uint32_t rid = range_from; rid < range_to; rid++) {
+ _block_rowids[nrows_read++] = rid;
+ }
+ } else {
+ nrows_read += rows_to_read;
+ }
+ } while (nrows_read < nrows_read_limit);
+ return Status::OK();
+}
+
+void SegmentIterator::_evaluate_vectorization_predicate(uint16_t* sel_rowid_idx, uint16_t& selected_size) {
+ uint16_t new_size = 0;
+ if (_vec_pred_column_ids.size() == 0) {
+ for (uint32_t i = 0; i < selected_size; ++i) {
+ sel_rowid_idx[new_size++] = i;
+ }
+ return;
+ }
+
+ uint16_t original_size = selected_size;
+ bool ret_flags[selected_size];
+ memset(ret_flags, 1, selected_size);
+ _pre_eval_block_predicate->evaluate_vec(_current_return_columns, selected_size, ret_flags);
+
+ for (uint32_t i = 0; i < selected_size; ++i) {
+ if (ret_flags[i]) {
+ sel_rowid_idx[new_size++] = i;
+ }
+ }
+
+ _opts.stats->rows_vec_cond_filtered += original_size - new_size;
+ selected_size = new_size;
+}
+
+void SegmentIterator::_evaluate_short_circuit_predicate(uint16_t* vec_sel_rowid_idx, uint16_t* selected_size_ptr) {
+ if (_short_cir_pred_column_ids.size() == 0) {
+ return;
+ }
+
+ for (auto column_predicate : _short_cir_eval_predicate) {
+ auto column_id = column_predicate->column_id();
+ auto& short_cir_column = _current_return_columns[column_id];
+ column_predicate->evaluate(*short_cir_column, vec_sel_rowid_idx, selected_size_ptr);
+ }
+
+ // evaluate delete condition
+ _opts.delete_condition_predicates->evaluate(_current_return_columns, vec_sel_rowid_idx, selected_size_ptr);
+}
+
+void SegmentIterator::_read_columns_by_rowids(std::vector<ColumnId>& read_column_ids, std::vector<rowid_t>& rowid_vector,
+ uint16_t* sel_rowid_idx, size_t select_size, vectorized::MutableColumns* mutable_columns) {
+ size_t start_idx = 0;
+ while (start_idx < select_size) {
+ size_t end_idx = start_idx + 1;
+ while (end_idx < select_size && (rowid_vector[sel_rowid_idx[end_idx - 1]] == rowid_vector[sel_rowid_idx[end_idx]] - 1)) {
+ end_idx++;
+ }
+ size_t range = end_idx - start_idx;
+ _seek_columns(read_column_ids, rowid_vector[sel_rowid_idx[start_idx]]);
+ _read_columns(read_column_ids, *mutable_columns, range);
+ start_idx += range;
+ }
+}
+
Status SegmentIterator::next_batch(vectorized::Block* block) {
- //TODO
- return Status::NotSupported("not implement now");
+ bool is_mem_reuse = block->mem_reuse();
+ SCOPED_RAW_TIMER(&_opts.stats->block_load_ns);
+ if (UNLIKELY(!_inited)) {
+ RETURN_IF_ERROR(_init(true));
+ _inited = true;
+ if (_vec_pred_column_ids.size() > 0 || _short_cir_pred_column_ids.size() > 0) {
+ _block_rowids.reserve(_opts.block_row_max);
+ }
+ _current_return_columns.resize(_schema.columns().size());
+ for (size_t i = 0; i < _schema.num_column_ids(); i++) {
+ auto cid = _schema.column_ids()[i];
+ if (_is_pred_column[cid]) {
+ auto& column_desc = _schema.columns()[cid];
+ _current_return_columns[cid] = Schema::get_predicate_column_nullable_ptr(column_desc->type(), column_desc->is_nullable());
+ _current_return_columns[cid]->reserve(_opts.block_row_max);
+ }
+ }
+ }
+
+ _init_current_block(block, _current_return_columns);
+
+ uint32_t nrows_read = 0;
+ uint32_t nrows_read_limit = _opts.block_row_max;
+ _read_columns_by_index(nrows_read_limit, nrows_read, _col_predicates.size() > 0);
+
+ _opts.stats->blocks_load += 1;
+ _opts.stats->raw_rows_read += nrows_read;
+
+ if (nrows_read == 0) {
+ for (int i = 0; i < _schema.num_column_ids(); i++) {
+ auto cid = _schema.column_ids()[i];
+ // todo(wb) abstract make column where
+ if (!_is_pred_column[cid]) { // non-predicate
+ block->replace_by_position(i, std::move(_current_return_columns[cid]));
+ } else { // predicate
+ if (!is_mem_reuse) {
+ auto* column_desc = _schema.columns()[cid];
+ auto data_type = Schema::get_data_type_ptr(column_desc->type());
+ block->replace_by_position(i, data_type->create_column());
+ }
+ }
+ // not sure whether block is clear before enter segmentIter, so clear it here.
+ if (is_mem_reuse) {
+ block->clear_column_data();
+ }
+ }
+ return Status::EndOfFile("no more data in segment");
+ }
+
+ // when no predicate(include delete condition) is provided, output column directly
+ if (_vec_pred_column_ids.size() == 0 && _short_cir_pred_column_ids.size() == 0) {
+ _output_non_pred_columns(block, is_mem_reuse);
+ } else { // need predicate evaluation
+ uint16_t selected_size = nrows_read;
+ uint16_t sel_rowid_idx[selected_size];
+
+ // step 1: evaluate vectorization predicate
+ _evaluate_vectorization_predicate(sel_rowid_idx, selected_size);
+
+ // When predicate column and no-predicate column are both basic type, lazy materialization is eliminate
+ // So output block directly after vecorization evaluation
+ if (_is_all_column_basic_type) {
+ _output_column_by_sel_idx(block, _first_read_column_ids, sel_rowid_idx, selected_size, is_mem_reuse);
+ return Status::OK();
+ }
+
+ // step 2: evaluate short ciruit predicate
+ // todo(wb) research whether need to read short predicate after vectorization evaluation
+ // to reduce cost of read short circuit columns.
+ // In SSB test, it make no difference; So need more scenarios to test
+ _evaluate_short_circuit_predicate(sel_rowid_idx, &selected_size);
+
+ // step3: read non_predicate column
+ if (_non_predicate_columns.size() != 0) {
+ _read_columns_by_rowids(_non_predicate_columns, _block_rowids, sel_rowid_idx, selected_size, &_current_return_columns);
+ }
+
+ // step4: output columns
+ // 4.1 output non-predicate column
+ _output_non_pred_columns(block, is_mem_reuse);
+
+ // 4.2 output short circuit predicate column
+ _output_column_by_sel_idx(block, _short_cir_pred_column_ids, sel_rowid_idx, selected_size, is_mem_reuse);
+ // 4.3 output vectorizatioin predicate column
+ _output_column_by_sel_idx(block, _vec_pred_column_ids, sel_rowid_idx, selected_size, is_mem_reuse);
+
+ }
+
+ return Status::OK();
}
} // namespace segment_v2
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.h b/be/src/olap/rowset/segment_v2/segment_iterator.h
index fabefd5..c06077e 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.h
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.h
@@ -60,7 +60,7 @@ public:
uint64_t data_id() const { return _segment->id(); }
private:
- Status _init();
+ Status _init(bool is_vec = false);
Status _init_return_column_iterators();
Status _init_bitmap_index_iterators();
@@ -78,6 +78,7 @@ private:
Status _apply_bitmap_index();
void _init_lazy_materialization();
+ void _vec_init_lazy_materialization();
uint32_t segment_id() const { return _segment->id(); }
uint32_t num_rows() const { return _segment->num_rows(); }
@@ -87,6 +88,17 @@ private:
Status _read_columns(const std::vector<ColumnId>& column_ids, RowBlockV2* block,
size_t row_offset, size_t nrows);
+ // for vectorization implementation
+ Status _read_columns(const std::vector<ColumnId>& column_ids, vectorized::MutableColumns& column_block, size_t nrows);
+ Status _read_columns_by_index(uint32_t nrows_read_limit, uint32_t& nrows_read, bool set_block_rowid);
+ void _init_current_block(vectorized::Block* block, std::vector<vectorized::MutableColumnPtr>& non_pred_vector);
+ void _evaluate_vectorization_predicate(uint16_t* sel_rowid_idx, uint16_t& selected_size);
+ void _evaluate_short_circuit_predicate(uint16_t* sel_rowid_idx, uint16_t* selected_size);
+ void _output_non_pred_columns(vectorized::Block* block, bool is_block_mem_reuse);
+ void _output_column_by_sel_idx(vectorized::Block* block, std::vector<ColumnId> columnids, uint16_t* sel_rowid_idx, uint16_t select_size, bool is_block_mem_reuse);
+ void _read_columns_by_rowids(std::vector<ColumnId>& read_column_ids, std::vector<rowid_t>& rowid_vector,
+ uint16_t* sel_rowid_idx, size_t select_size, vectorized::MutableColumns* mutable_columns);
+
private:
class BitmapRangeIterator;
@@ -116,6 +128,21 @@ private:
// could be a local variable of next_batch(), kept here to reuse vector memory
std::vector<rowid_t> _block_rowids;
+ // fields for vectorization execution
+ bool _is_all_column_basic_type;
+ std::vector<ColumnId> _vec_pred_column_ids; // keep columnId of columns for vectorized predicate evaluation
+ std::vector<ColumnId> _short_cir_pred_column_ids; // keep columnId of columns for short circuit predicate evaluation
+ vector<bool> _is_pred_column; // columns hold by segmentIter
+ vectorized::MutableColumns _current_return_columns;
+ AndBlockColumnPredicate* _pre_eval_block_predicate = nullptr;
+ std::vector<ColumnPredicate*> _short_cir_eval_predicate;
+ // when lazy materialization is enable, segmentIter need to read data at least twice
+ // first, read predicate columns by various index
+ // second, read non-predicate columns
+ // so we need a field to stand for columns first time to read
+ vector<ColumnId> _first_read_column_ids;
+ vector<int> _schema_block_id_map; // map from schema column id to column idx in Block
+
// the actual init process is delayed to the first call to next_batch()
bool _inited;
diff --git a/be/src/olap/schema.cpp b/be/src/olap/schema.cpp
index d588b58..f5a0900 100644
--- a/be/src/olap/schema.cpp
+++ b/be/src/olap/schema.cpp
@@ -18,6 +18,11 @@
#include "olap/schema.h"
#include "olap/row_block2.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_complex.h"
+#include "vec/columns/predicate_column.h"
+#include "vec/core/types.h"
+#include "olap/uint24.h"
namespace doris {
@@ -105,6 +110,9 @@ Schema::~Schema() {
vectorized::DataTypePtr Schema::get_data_type_ptr(FieldType type) {
switch (type) {
+ case OLAP_FIELD_TYPE_BOOL:
+ return std::make_shared<vectorized::DataTypeUInt8>();
+
case OLAP_FIELD_TYPE_TINYINT:
return std::make_shared<vectorized::DataTypeInt8>();
@@ -135,7 +143,11 @@ vectorized::DataTypePtr Schema::get_data_type_ptr(FieldType type) {
case OLAP_FIELD_TYPE_CHAR:
case OLAP_FIELD_TYPE_VARCHAR:
case OLAP_FIELD_TYPE_HLL:
+ case OLAP_FIELD_TYPE_STRING:
return std::make_shared<vectorized::DataTypeString>();
+
+ case TYPE_OBJECT:
+ return std::make_shared<vectorized::DataTypeBitMap>();
case OLAP_FIELD_TYPE_DECIMAL:
return std::make_shared<vectorized::DataTypeDecimal<vectorized::Decimal128>>(27, 9);
@@ -147,4 +159,58 @@ vectorized::DataTypePtr Schema::get_data_type_ptr(FieldType type) {
return nullptr;
}
+vectorized::IColumn::MutablePtr Schema::get_predicate_column_nullable_ptr(FieldType type, bool is_null) {
+ vectorized::IColumn::MutablePtr ptr = Schema::get_predicate_column_ptr(type);
+ if (is_null) {
+ return doris::vectorized::ColumnNullable::create(std::move(ptr), doris::vectorized::ColumnUInt8::create());
+ }
+ return ptr;
+}
+
+vectorized::IColumn::MutablePtr Schema::get_predicate_column_ptr(FieldType type) {
+ switch (type) {
+ case OLAP_FIELD_TYPE_BOOL:
+ return doris::vectorized::PredicateColumnType<bool>::create();;
+ case OLAP_FIELD_TYPE_TINYINT:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Int8>::create();
+
+ case OLAP_FIELD_TYPE_SMALLINT:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Int16>::create();
+
+ case OLAP_FIELD_TYPE_INT:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Int32>::create();
+
+ case OLAP_FIELD_TYPE_FLOAT:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Float32>::create();
+
+ case OLAP_FIELD_TYPE_DOUBLE:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Float64>::create();
+
+ case OLAP_FIELD_TYPE_BIGINT:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Int64>::create();
+
+ case OLAP_FIELD_TYPE_LARGEINT:
+ return doris::vectorized::PredicateColumnType<doris::vectorized::Int128>::create();
+
+ case OLAP_FIELD_TYPE_DATE:
+ return doris::vectorized::PredicateColumnType<uint24_t>::create();
+
+ case OLAP_FIELD_TYPE_DATETIME:
+ return doris::vectorized::PredicateColumnType<uint64_t>::create();
+
+ case OLAP_FIELD_TYPE_CHAR:
+ case OLAP_FIELD_TYPE_VARCHAR:
+ case OLAP_FIELD_TYPE_STRING:
+ return doris::vectorized::PredicateColumnType<StringValue>::create();
+
+ case OLAP_FIELD_TYPE_DECIMAL:
+ return doris::vectorized::PredicateColumnType<decimal12_t>::create();
+
+ default:
+ DCHECK(false);
+ }
+ // For llvm complain
+ return nullptr;
+}
+
} // namespace doris
diff --git a/be/src/olap/schema.h b/be/src/olap/schema.h
index 3fd90bf..2596f97 100644
--- a/be/src/olap/schema.h
+++ b/be/src/olap/schema.h
@@ -102,6 +102,10 @@ public:
static vectorized::DataTypePtr get_data_type_ptr(FieldType type);
+ static vectorized::IColumn::MutablePtr get_predicate_column_ptr(FieldType type);
+
+ static vectorized::IColumn::MutablePtr get_predicate_column_nullable_ptr(FieldType type, bool is_null = false);
+
const std::vector<Field*>& columns() const { return _cols; }
const Field* column(ColumnId cid) const { return _cols[cid]; }
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index 5e41028..a869a65 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -215,7 +215,7 @@ public:
* used by lazy materialization to filter column by selected rowids
*/
virtual Ptr filter_by_selector(const uint16_t* sel, size_t sel_size, Ptr* ptr = nullptr) {
- return nullptr;
+ LOG(FATAL) << "column not support filter_by_selector";
};
/// Permutes elements using specified permutation. Is used in sortings.
@@ -333,6 +333,8 @@ public:
/// True if column contains something nullable inside. It's true for ColumnNullable, can be true or false for ColumnConst, etc.
virtual bool is_nullable() const { return false; }
+ virtual bool is_bitmap() const { return false; }
+
// true iff column has null element
virtual bool has_null() const { return false; }
@@ -414,10 +416,15 @@ public:
// only used in ColumnNullable replace_column_data
virtual void replace_column_data_default(size_t self_row = 0) = 0;
- bool is_date_type() { return is_date; }
+ virtual bool is_date_type() { return is_date; }
+ virtual bool is_datetime_type() { return is_date_time; }
+
+ virtual void set_date_type() { is_date = true; }
+ virtual void set_datetime_type() { is_date_time = true; }
- // todo(wb): a temporary implemention, need refactor here
+ // todo(wb): a temporary implemention, need re-abstract here
bool is_date = false;
+ bool is_date_time = false;
protected:
/// Template is to devirtualize calls to insert_from method.
diff --git a/be/src/vec/columns/column_complex.h b/be/src/vec/columns/column_complex.h
index 63a120a..296f94b 100644
--- a/be/src/vec/columns/column_complex.h
+++ b/be/src/vec/columns/column_complex.h
@@ -45,6 +45,8 @@ public:
bool is_numeric() const override { return false; }
+ bool is_bitmap() const override { return std::is_same_v<T, BitmapValue>; }
+
size_t size() const override { return data.size(); }
StringRef get_data_at(size_t n) const override {
diff --git a/be/src/vec/columns/column_nullable.cpp b/be/src/vec/columns/column_nullable.cpp
index bee6909..bf4bb44 100644
--- a/be/src/vec/columns/column_nullable.cpp
+++ b/be/src/vec/columns/column_nullable.cpp
@@ -177,9 +177,18 @@ ColumnPtr ColumnNullable::filter(const Filter& filt, ssize_t result_size_hint) c
}
ColumnPtr ColumnNullable::filter_by_selector(const uint16_t* sel, size_t sel_size, ColumnPtr* ptr) {
- ColumnPtr filtered_data = get_nested_column().filter_by_selector(sel, sel_size, ptr);
- ColumnPtr filtered_null_map = get_null_map_column().filter_by_selector(sel, sel_size, ptr);
- return ColumnNullable::create(filtered_data, filtered_null_map);
+ if (ptr != nullptr) {
+ const ColumnNullable* nullable_col_ptr = reinterpret_cast<const ColumnNullable*>((*ptr).get());
+ ColumnPtr nest_col_ptr = nullable_col_ptr->nested_column;
+ ColumnPtr null_map_ptr = nullable_col_ptr->null_map;
+ get_nested_column().filter_by_selector(sel, sel_size, &nest_col_ptr);
+ get_null_map_column().filter_by_selector(sel, sel_size, &null_map_ptr);
+ return *ptr;
+ } else {
+ ColumnPtr filtered_data = get_nested_column().filter_by_selector(sel, sel_size);
+ ColumnPtr filtered_null_map = get_null_map_column().filter_by_selector(sel, sel_size);
+ return ColumnNullable::create(filtered_data, filtered_null_map);
+ }
}
ColumnPtr ColumnNullable::permute(const Permutation& perm, size_t limit) const {
diff --git a/be/src/vec/columns/column_nullable.h b/be/src/vec/columns/column_nullable.h
index 8f47777..fb6aa8f 100644
--- a/be/src/vec/columns/column_nullable.h
+++ b/be/src/vec/columns/column_nullable.h
@@ -130,7 +130,13 @@ public:
return false;
}
+ bool is_date_type() override { return get_nested_column().is_date_type(); }
+ bool is_datetime_type() override { return get_nested_column().is_datetime_type(); }
+ void set_date_type() override { get_nested_column().set_date_type(); }
+ void set_datetime_type() override { get_nested_column().set_datetime_type(); }
+
bool is_nullable() const override { return true; }
+ bool is_bitmap() const override { return get_nested_column().is_bitmap(); }
bool is_column_decimal() const override { return get_nested_column().is_column_decimal(); }
bool is_column_string() const override { return get_nested_column().is_column_string(); }
bool is_fixed_and_contiguous() const override { return false; }
diff --git a/be/src/vec/columns/column_vector.h b/be/src/vec/columns/column_vector.h
index 5323f60..6626229 100644
--- a/be/src/vec/columns/column_vector.h
+++ b/be/src/vec/columns/column_vector.h
@@ -119,6 +119,13 @@ private:
/// Sugar constructor.
ColumnVector(std::initializer_list<T> il) : data {il} {}
+ void insert_res_column(const uint16_t* sel, size_t sel_size, vectorized::ColumnVector<T>* res_ptr) {
+ for (size_t i = 0; i < sel_size; i++) {
+ T* val_ptr = &data[sel[i]];
+ res_ptr->insert_data((char*)val_ptr, 0);
+ }
+ }
+
public:
bool is_numeric() const override { return IsNumber<T>; }
@@ -195,6 +202,25 @@ public:
ColumnPtr filter(const IColumn::Filter& filt, ssize_t result_size_hint) const override;
+ // note(wb) this method is only used in storage layer now
+ ColumnPtr filter_by_selector(const uint16_t* sel, size_t sel_size, ColumnPtr* ptr = nullptr) override {
+ if (ptr == nullptr) {
+ auto res_ptr = vectorized::ColumnVector<T>::create();
+ if (sel_size == 0) {
+ return res_ptr;
+ }
+ insert_res_column(sel, sel_size, res_ptr.get());
+ return res_ptr;
+ } else {
+ auto res_ptr = (*std::move(*ptr)).assume_mutable();
+ if (sel_size == 0) {
+ return res_ptr;
+ }
+ insert_res_column(sel, sel_size, reinterpret_cast<vectorized::ColumnVector<T>*>(res_ptr.get()));
+ return *ptr;
+ }
+ }
+
ColumnPtr permute(const IColumn::Permutation& perm, size_t limit) const override;
// ColumnPtr index(const IColumn & indexes, size_t limit) const override;
diff --git a/be/src/vec/columns/predicate_column.h b/be/src/vec/columns/predicate_column.h
index c8f8627..8095cff 100644
--- a/be/src/vec/columns/predicate_column.h
+++ b/be/src/vec/columns/predicate_column.h
@@ -97,6 +97,14 @@ private:
}
}
+ void insert_byte_to_res_column(const uint16_t* sel, size_t sel_size, vectorized::IColumn* res_ptr) {
+ for (size_t i = 0; i < sel_size; i++) {
+ uint16_t n = sel[i];
+ char* ch_val = reinterpret_cast<char*>(&data[n]);
+ res_ptr->insert_data(ch_val, 0);
+ }
+ }
+
template <typename Y>
ColumnPtr filter_default_type_by_selector(const uint16_t* sel, size_t sel_size, ColumnPtr* ptr = nullptr) {
static_assert(std::is_same_v<T, Y>);
@@ -371,6 +379,23 @@ public:
return *ptr;
}
+ ColumnPtr filter_bool_by_selector(const uint16_t* sel, size_t sel_size, ColumnPtr* ptr = nullptr) {
+ if (ptr == nullptr) {
+ auto res = vectorized::ColumnVector<vectorized::UInt8>::create();
+ if (sel_size == 0) {
+ return res;
+ }
+ res->reserve(sel_size);
+ insert_byte_to_res_column(sel, sel_size, res.get());
+ } else {
+ if (sel_size != 0) {
+ MutableColumnPtr ptr_res = (*std::move(*ptr)).assume_mutable();
+ insert_byte_to_res_column(sel, sel_size, ptr_res.get());
+ }
+ }
+ return *ptr;
+ }
+
//todo(wb) need refactor this method, using return status to check unexpect args instead of LOG(FATAL)
ColumnPtr filter_by_selector(const uint16_t* sel, size_t sel_size, ColumnPtr* ptr = nullptr) override {
if constexpr (std::is_same_v<T, StringValue>) {
@@ -396,7 +421,7 @@ public:
} else if constexpr (std::is_same_v<T, doris::vectorized::Int128>) {
return filter_default_type_by_selector<doris::vectorized::Int128>(sel, sel_size, ptr);
} else if (std::is_same_v<T, bool>) {
- LOG(FATAL) << "bool will be support later";
+ return filter_bool_by_selector(sel, sel_size, ptr);
} else {
LOG(FATAL) << "unexpected type in predicate column";
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 17/33: [Feature][Vectorized] Support String in vec exe engine (#7670)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 1fc1c7005dccabdadeb0a5d2cd0cf0aa4871a3a8
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Mon Jan 10 20:27:45 2022 +0800
[Feature][Vectorized] Support String in vec exe engine (#7670)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/olap/olap_define.h | 3 ++-
be/src/olap/row_block2.cpp | 26 ++++++++++++++++++++--
be/src/olap/row_block2.h | 2 +-
be/src/olap/rowset/beta_rowset_reader.cpp | 6 ++++-
be/src/vec/exec/vunion_node.cpp | 9 ++++++++
.../apache/doris/rewrite/FoldConstantsRule.java | 7 ------
6 files changed, 41 insertions(+), 12 deletions(-)
diff --git a/be/src/olap/olap_define.h b/be/src/olap/olap_define.h
index c2d4b7f..a9ac731 100644
--- a/be/src/olap/olap_define.h
+++ b/be/src/olap/olap_define.h
@@ -384,7 +384,8 @@ enum OLAPStatus {
OLAP_ERR_ROWSET_LOAD_FAILED = -3109,
OLAP_ERR_ROWSET_READER_INIT = -3110,
OLAP_ERR_ROWSET_READ_FAILED = -3111,
- OLAP_ERR_ROWSET_INVALID_STATE_TRANSITION = -3112
+ OLAP_ERR_ROWSET_INVALID_STATE_TRANSITION = -3112,
+ OLAP_ERR_STRING_OVERFLOW_IN_VEC_ENGINE = -3113
};
enum ColumnFamilyIndex {
diff --git a/be/src/olap/row_block2.cpp b/be/src/olap/row_block2.cpp
index 26b58ca..877f6a2 100644
--- a/be/src/olap/row_block2.cpp
+++ b/be/src/olap/row_block2.cpp
@@ -95,7 +95,9 @@ Status RowBlockV2::convert_to_row_block(RowCursor* helper, RowBlock* dst) {
return Status::OK();
}
-void RowBlockV2::_copy_data_to_column(int cid, doris::vectorized::MutableColumnPtr& origin_column) {
+Status RowBlockV2::_copy_data_to_column(int cid, doris::vectorized::MutableColumnPtr& origin_column) {
+ constexpr auto MAX_SIZE_OF_VEC_STRING = 1024l * 1024;
+
auto* column = origin_column.get();
bool nullable_mark_array[_selected_size];
@@ -170,6 +172,24 @@ void RowBlockV2::_copy_data_to_column(int cid, doris::vectorized::MutableColumnP
}
break;
}
+ case OLAP_FIELD_TYPE_STRING: {
+ auto column_string = assert_cast<vectorized::ColumnString*>(column);
+
+ for (uint16_t j = 0; j < _selected_size; ++j) {
+ if (!nullable_mark_array[j]) {
+ uint16_t row_idx = _selection_vector[j];
+ auto slice = reinterpret_cast<const Slice*>(column_block(cid).cell_ptr(row_idx));
+ if (LIKELY(slice->size <= MAX_SIZE_OF_VEC_STRING)) {
+ column_string->insert_data(slice->data, slice->size);
+ } else {
+ return Status::NotSupported("Not support string len over than 1MB in vec engine.");
+ }
+ } else {
+ column_string->insert_default();
+ }
+ }
+ break;
+ }
case OLAP_FIELD_TYPE_CHAR: {
auto column_string = assert_cast<vectorized::ColumnString*>(column);
@@ -286,13 +306,15 @@ void RowBlockV2::_copy_data_to_column(int cid, doris::vectorized::MutableColumnP
DCHECK(false) << "Invalid type in RowBlockV2:" << _schema.column(cid)->type();
}
}
+
+ return Status::OK();
}
Status RowBlockV2::convert_to_vec_block(vectorized::Block* block) {
for (int i = 0; i < _schema.column_ids().size(); ++i) {
auto cid = _schema.column_ids()[i];
auto column = (*std::move(block->get_by_position(i).column)).assume_mutable();
- _copy_data_to_column(cid, column);
+ RETURN_IF_ERROR(_copy_data_to_column(cid, column));
}
_pool->clear();
return Status::OK();
diff --git a/be/src/olap/row_block2.h b/be/src/olap/row_block2.h
index cdbf428..b98ab95 100644
--- a/be/src/olap/row_block2.h
+++ b/be/src/olap/row_block2.h
@@ -109,7 +109,7 @@ public:
std::string debug_string();
private:
- void _copy_data_to_column(int cid, vectorized::MutableColumnPtr& mutable_column_ptr);
+ Status _copy_data_to_column(int cid, vectorized::MutableColumnPtr& mutable_column_ptr);
Schema _schema;
size_t _capacity;
diff --git a/be/src/olap/rowset/beta_rowset_reader.cpp b/be/src/olap/rowset/beta_rowset_reader.cpp
index 459f3ca..4d35f2f 100644
--- a/be/src/olap/rowset/beta_rowset_reader.cpp
+++ b/be/src/olap/rowset/beta_rowset_reader.cpp
@@ -204,7 +204,11 @@ OLAPStatus BetaRowsetReader::next_block(vectorized::Block* block) {
{
SCOPED_RAW_TIMER(&_stats->block_convert_ns);
- _input_block->convert_to_vec_block(block);
+ auto s = _input_block->convert_to_vec_block(block);
+ if (UNLIKELY(!s.ok())) {
+ LOG(WARNING) << "failed to read next block: " << s.to_string();
+ return OLAP_ERR_STRING_OVERFLOW_IN_VEC_ENGINE;
+ }
}
is_first = false;
} while (block->rows() < _context->runtime_state->batch_size()); // here we should keep block.rows() < batch_size
diff --git a/be/src/vec/exec/vunion_node.cpp b/be/src/vec/exec/vunion_node.cpp
index 1fa4da4..c05b3ef 100644
--- a/be/src/vec/exec/vunion_node.cpp
+++ b/be/src/vec/exec/vunion_node.cpp
@@ -195,6 +195,15 @@ Status VUnionNode::get_next_const(RuntimeState* state, Block* block) {
if (!mem_reuse) {
block->swap(mblock.to_block());
}
+
+ // some insert query like "insert into string_test select 1, repeat('a', 1024 * 1024);"
+ // the const expr will be in output expr cause the union node return a empty block. so here we
+ // need add one row to make sure the union node exec const expr return at least one row
+ if (block->rows() == 0) {
+ block->insert({vectorized::ColumnUInt8::create(1),
+ std::make_shared<vectorized::DataTypeUInt8>(), ""});
+ }
+
return Status::OK();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
index 628740a..8f5137f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
@@ -27,7 +27,6 @@ import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.NullLiteral;
import org.apache.doris.analysis.SysVariableDesc;
import org.apache.doris.catalog.Catalog;
-import org.apache.doris.catalog.Function;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
@@ -195,12 +194,6 @@ public class FoldConstantsRule implements ExprRewriteRule {
Analyzer analyzer, Map<String, Expr> sysVarMap, Map<String, Expr> infoFnMap)
throws AnalysisException {
if (expr.isConstant()) {
- if (VectorizedUtil.isVectorized()) {
- Function fn = expr.getFn();
- if (fn != null && (fn.functionName().equalsIgnoreCase("curtime") ||
- fn.functionName().equalsIgnoreCase("current_time")))
- return;
- }
// Do not constant fold cast(null as dataType) because we cannot preserve the
// cast-to-types and that can lead to query failures, e.g., CTAS
if (expr instanceof CastExpr) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 16/33: [Vectorized] [Function] Support do not fold constant at vectorized (#7668)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit acb63c749cff8b323cf9efca8cf47ddb171aecd0
Author: Pxl <95...@qq.com>
AuthorDate: Mon Jan 10 10:52:44 2022 +0800
[Vectorized] [Function] Support do not fold constant at vectorized (#7668)
---
.../org/apache/doris/analysis/ArithmeticExpr.java | 30 ++++++++++
.../java/org/apache/doris/rewrite/FEFunctions.java | 64 ++++++++++++++++++++--
2 files changed, 88 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ArithmeticExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ArithmeticExpr.java
index 50012b7..79a1ffa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ArithmeticExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ArithmeticExpr.java
@@ -267,6 +267,33 @@ public class ArithmeticExpr extends Expr {
}
}
+ private boolean castIfHaveSameType(Type t1, Type t2, Type target) throws AnalysisException {
+ if (t1 == target || t2 == target) {
+ castChild(target, 0);
+ castChild(target, 1);
+ return true;
+ }
+ return false;
+ }
+
+ private void castUpperInteger(Type t1, Type t2) throws AnalysisException {
+ if (!t1.isIntegerType() || !t2.isIntegerType()) {
+ return;
+ }
+ if (castIfHaveSameType(t1, t2, Type.BIGINT)) {
+ return;
+ }
+ if (castIfHaveSameType(t1, t2, Type.INT)) {
+ return;
+ }
+ if (castIfHaveSameType(t1, t2, Type.SMALLINT)) {
+ return;
+ }
+ if (castIfHaveSameType(t1, t2, Type.TINYINT)) {
+ return;
+ }
+ }
+
@Override
public void analyzeImpl(Analyzer analyzer) throws AnalysisException {
if (VectorizedUtil.isVectorized()) {
@@ -320,6 +347,9 @@ public class ArithmeticExpr extends Expr {
if (t1.isDecimalV2() || t2.isDecimalV2()) {
castBinaryOp(findCommonType(t1, t2));
}
+ if (isConstant()) {
+ castUpperInteger(t1, t2);
+ }
case MOD:
if (t1.isDecimalV2() || t2.isDecimalV2()) {
castBinaryOp(findCommonType(t1, t2));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
index 26ca3f7..0bcbfb6 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
@@ -350,12 +350,30 @@ public class FEFunctions {
* Arithmetic function
*/
- @FEFunction(name = "add", argTypes = { "BIGINT", "BIGINT" }, returnType = "BIGINT")
+ @FEFunction(name = "add", argTypes = { "TINYINT", "TINYINT" }, returnType = "SMALLINT")
+ public static IntLiteral addTinyint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.addExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.SMALLINT);
+ }
+
+ @FEFunction(name = "add", argTypes = { "SMALLINT", "SMALLINT" }, returnType = "INT")
+ public static IntLiteral addSmallint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.addExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.INT);
+ }
+
+ @FEFunction(name = "add", argTypes = { "INT", "INT" }, returnType = "BIGINT")
public static IntLiteral addInt(LiteralExpr first, LiteralExpr second) throws AnalysisException {
long result = Math.addExact(first.getLongValue(), second.getLongValue());
return new IntLiteral(result, Type.BIGINT);
}
+ @FEFunction(name = "add", argTypes = { "BIGINT", "BIGINT" }, returnType = "BIGINT")
+ public static IntLiteral addBigint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.addExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.BIGINT);
+ }
+
@FEFunction(name = "add", argTypes = { "DOUBLE", "DOUBLE" }, returnType = "DOUBLE")
public static FloatLiteral addDouble(LiteralExpr first, LiteralExpr second) throws AnalysisException {
double result = first.getDoubleValue() + second.getDoubleValue();
@@ -379,12 +397,30 @@ public class FEFunctions {
return new LargeIntLiteral(result.toString());
}
- @FEFunction(name = "subtract", argTypes = { "BIGINT", "BIGINT" }, returnType = "BIGINT")
+ @FEFunction(name = "subtract", argTypes = { "TINYINT", "TINYINT" }, returnType = "SMALLINT")
+ public static IntLiteral subtractTinyint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.subtractExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.SMALLINT);
+ }
+
+ @FEFunction(name = "subtract", argTypes = { "SMALLINT", "SMALLINT" }, returnType = "INT")
+ public static IntLiteral subtractSmallint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.subtractExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.INT);
+ }
+
+ @FEFunction(name = "subtract", argTypes = { "INT", "INT" }, returnType = "BIGINT")
public static IntLiteral subtractInt(LiteralExpr first, LiteralExpr second) throws AnalysisException {
long result = Math.subtractExact(first.getLongValue(), second.getLongValue());
return new IntLiteral(result, Type.BIGINT);
}
+ @FEFunction(name = "subtract", argTypes = { "BIGINT", "BIGINT" }, returnType = "BIGINT")
+ public static IntLiteral subtractBigint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.subtractExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.BIGINT);
+ }
+
@FEFunction(name = "subtract", argTypes = { "DOUBLE", "DOUBLE" }, returnType = "DOUBLE")
public static FloatLiteral subtractDouble(LiteralExpr first, LiteralExpr second) throws AnalysisException {
double result = first.getDoubleValue() - second.getDoubleValue();
@@ -408,11 +444,27 @@ public class FEFunctions {
return new LargeIntLiteral(result.toString());
}
- @FEFunction(name = "multiply", argTypes = { "BIGINT", "BIGINT" }, returnType = "BIGINT")
+ @FEFunction(name = "multiply", argTypes = { "TINYINT", "TINYINT" }, returnType = "SMALLINT")
+ public static IntLiteral multiplyTinyint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.multiplyExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.SMALLINT);
+ }
+
+ @FEFunction(name = "multiply", argTypes = { "SMALLINT", "SMALLINT" }, returnType = "INT")
+ public static IntLiteral multiplySmallint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.multiplyExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.INT);
+ }
+
+ @FEFunction(name = "multiply", argTypes = { "INT", "INT" }, returnType = "BIGINT")
public static IntLiteral multiplyInt(LiteralExpr first, LiteralExpr second) throws AnalysisException {
- long left = first.getLongValue();
- long right = second.getLongValue();
- long result = Math.multiplyExact(left, right);
+ long result = Math.multiplyExact(first.getLongValue(), second.getLongValue());
+ return new IntLiteral(result, Type.BIGINT);
+ }
+
+ @FEFunction(name = "multiply", argTypes = { "BIGINT", "BIGINT" }, returnType = "BIGINT")
+ public static IntLiteral multiplyBigint(LiteralExpr first, LiteralExpr second) throws AnalysisException {
+ long result = Math.multiplyExact(first.getLongValue(), second.getLongValue());
return new IntLiteral(result, Type.BIGINT);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 29/33: [Vectorized][feature](planner)(executor) Support grouping sets rollup cube (#7601)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8760e644f4086809f214f0d105f0ae4492024ca6
Author: anneji-dev <85...@users.noreply.github.com>
AuthorDate: Mon Jan 17 14:09:15 2022 +0800
[Vectorized][feature](planner)(executor) Support grouping sets rollup cube (#7601)
---
be/src/exec/exec_node.cpp | 8 +-
be/src/exec/repeat_node.h | 2 +-
be/src/vec/CMakeLists.txt | 2 +
be/src/vec/exec/vrepeat_node.cpp | 245 +++++++++++++++++++++
be/src/vec/exec/vrepeat_node.h | 56 +++++
be/src/vec/functions/function_grouping.cpp | 25 +++
be/src/vec/functions/function_grouping.h | 90 ++++++++
be/src/vec/functions/simple_function_factory.h | 2 +
.../apache/doris/planner/SingleNodePlanner.java | 10 +
gensrc/script/doris_builtins_functions.py | 4 +-
10 files changed, 440 insertions(+), 4 deletions(-)
diff --git a/be/src/exec/exec_node.cpp b/be/src/exec/exec_node.cpp
index 4582f89..97c3259 100644
--- a/be/src/exec/exec_node.cpp
+++ b/be/src/exec/exec_node.cpp
@@ -82,6 +82,7 @@
#include "vec/exprs/vexpr.h"
#include "vec/exec/vempty_set_node.h"
#include "vec/exec/vschema_scan_node.h"
+#include "vec/exec/vrepeat_node.h"
namespace doris {
const std::string ExecNode::ROW_THROUGHPUT_COUNTER = "RowsReturnedRate";
@@ -389,6 +390,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::SCHEMA_SCAN_NODE:
case TPlanNodeType::ANALYTIC_EVAL_NODE:
case TPlanNodeType::SELECT_NODE:
+ case TPlanNodeType::REPEAT_NODE:
break;
default: {
const auto& i = _TPlanNodeType_VALUES_TO_NAMES.find(tnode.node_type);
@@ -568,7 +570,11 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
return Status::OK();
case TPlanNodeType::REPEAT_NODE:
- *node = pool->add(new RepeatNode(pool, tnode, descs));
+ if (state->enable_vectorized_exec()) {
+ *node = pool->add(new vectorized::VRepeatNode(pool, tnode, descs));
+ } else {
+ *node = pool->add(new RepeatNode(pool, tnode, descs));
+ }
return Status::OK();
case TPlanNodeType::ASSERT_NUM_ROWS_NODE:
diff --git a/be/src/exec/repeat_node.h b/be/src/exec/repeat_node.h
index 01335d2..d9dce75 100644
--- a/be/src/exec/repeat_node.h
+++ b/be/src/exec/repeat_node.h
@@ -40,7 +40,7 @@ public:
protected:
virtual void debug_string(int indentation_level, std::stringstream* out) const override;
-private:
+protected:
Status get_repeated_batch(RowBatch* child_row_batch, int repeat_id_idx, RowBatch* row_batch);
// Slot id set used to indicate those slots need to set to null.
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 9c4d947..f737391 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -86,6 +86,7 @@ set(VEC_FILES
exec/vempty_set_node.cpp
exec/vanalytic_eval_node.cpp
exec/vassert_num_rows_node.cpp
+ exec/vrepeat_node.cpp
exec/join/vhash_join_node.cpp
exprs/vectorized_agg_fn.cpp
exprs/vectorized_fn_call.cpp
@@ -139,6 +140,7 @@ set(VEC_FILES
functions/function_date_or_datetime_computation.cpp
functions/function_date_or_datetime_to_string.cpp
functions/function_datetime_string_to_string.cpp
+ functions/function_grouping.cpp
olap/vgeneric_iterators.cpp
olap/vcollect_iterator.cpp
olap/block_reader.cpp
diff --git a/be/src/vec/exec/vrepeat_node.cpp b/be/src/vec/exec/vrepeat_node.cpp
new file mode 100644
index 0000000..dd8bb28
--- /dev/null
+++ b/be/src/vec/exec/vrepeat_node.cpp
@@ -0,0 +1,245 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/exec/vrepeat_node.h"
+#include "exprs/expr.h"
+#include "gutil/strings/join.h"
+#include "runtime/runtime_state.h"
+#include "util/runtime_profile.h"
+
+namespace doris::vectorized {
+VRepeatNode::VRepeatNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs)
+ : RepeatNode(pool, tnode, descs), _child_block(nullptr), _virtual_tuple_id(tnode.repeat_node.output_tuple_id) {}
+
+Status VRepeatNode::prepare(RuntimeState* state) {
+ VLOG_CRITICAL << "VRepeatNode::prepare";
+ SCOPED_TIMER(_runtime_profile->total_time_counter());
+ RETURN_IF_ERROR(RepeatNode::prepare(state));
+
+ // get current all output slots
+ for (const auto& tuple_desc : this->row_desc().tuple_descriptors()) {
+ for (const auto& slot_desc : tuple_desc->slots()) {
+ _output_slots.push_back(slot_desc);
+ }
+ }
+
+ // get all input slots
+ for (const auto& child_tuple_desc : child(0)->row_desc().tuple_descriptors()) {
+ for (const auto& child_slot_desc : child_tuple_desc->slots()) {
+ _child_slots.push_back(child_slot_desc);
+ }
+ }
+
+ _virtual_tuple_desc = state->desc_tbl().get_tuple_descriptor(_virtual_tuple_id);
+ if (_virtual_tuple_desc == NULL) {
+ return Status::InternalError("Failed to get virtual tuple descriptor.");
+ }
+
+ std::stringstream ss;
+ ss << "The output slots size " << _output_slots.size()
+ << " is not equal to the sum of child_slots_size " << _child_slots.size()
+ << ",virtual_slots_size " << _virtual_tuple_desc->slots().size();
+ if (_output_slots.size() != (_child_slots.size() + _virtual_tuple_desc->slots().size())) {
+ return Status::InternalError(ss.str());
+ }
+
+ _child_block.reset(new Block());
+
+ return Status::OK();
+}
+
+Status VRepeatNode::open(RuntimeState* state) {
+ VLOG_CRITICAL << "VRepeatNode::open";
+ SCOPED_TIMER(_runtime_profile->total_time_counter());
+ RETURN_IF_ERROR(RepeatNode::open(state));
+ return Status::OK();
+}
+
+Status VRepeatNode::get_repeated_block(Block* child_block, int repeat_id_idx, Block* output_block) {
+ VLOG_CRITICAL << "VRepeatNode::get_repeated_block";
+ DCHECK(child_block != nullptr);
+ DCHECK_EQ(output_block->rows(), 0);
+
+ size_t child_column_size = child_block->columns();
+ size_t column_size = _output_slots.size();
+ bool mem_reuse = output_block->mem_reuse();
+ DCHECK_EQ(child_column_size, _child_slots.size());
+ DCHECK_LT(child_column_size, column_size);
+ std::vector<vectorized::MutableColumnPtr> columns(column_size);
+ for (size_t i = 0; i < column_size; i++) {
+ if (mem_reuse) {
+ columns[i] = std::move(*output_block->get_by_position(i).column).mutate();
+ } else {
+ columns[i] = _output_slots[i]->get_empty_mutable_column();
+ }
+ }
+
+ /* Fill all slots according to child, for example:select tc1,tc2,sum(tc3) from t1 group by grouping sets((tc1),(tc2));
+ * insert into t1 values(1,2,1),(1,3,1),(2,1,1),(3,1,1);
+ * slot_id_set_list=[[0],[1]],repeat_id_idx=0,
+ * child_block 1,2,1 | 1,3,1 | 2,1,1 | 3,1,1
+ * output_block 1,null,1,1 | 1,null,1,1 | 2,nul,1,1 | 3,null,1,1
+ */
+ size_t cur_col = 0;
+ for (size_t i = 0; i < child_column_size; i++) {
+ const ColumnWithTypeAndName& src_column = child_block->get_by_position(i);
+
+ DCHECK_EQ(_child_slots[i]->type().type, _output_slots[cur_col]->type().type);
+ DCHECK_EQ(_child_slots[i]->col_name(), _output_slots[cur_col]->col_name());
+
+ std::set<SlotId>& repeat_ids = _slot_id_set_list[repeat_id_idx];
+ bool is_repeat_slot = _all_slot_ids.find(_output_slots[cur_col]->id()) != _all_slot_ids.end();
+ bool is_set_null_slot = repeat_ids.find(_output_slots[cur_col]->id()) == repeat_ids.end();
+
+ if (is_repeat_slot) {
+ DCHECK(_output_slots[cur_col]->is_nullable());
+ auto* nullable_column = reinterpret_cast<ColumnNullable *>(columns[cur_col].get());
+ auto& null_map = nullable_column->get_null_map_data();
+ auto* column_ptr = columns[cur_col].get();
+
+ // set slot null not in repeat_ids
+ if (is_set_null_slot) {
+ nullable_column->resize(src_column.column->size());
+ for (size_t j = 0; j < src_column.column->size(); ++j) {
+ nullable_column->insert_data(nullptr, 0);
+ }
+ } else {
+ if (!src_column.type->is_nullable()) {
+ for (size_t j = 0; j < src_column.column->size(); ++j) {
+ null_map.push_back(0);
+ }
+ column_ptr = &nullable_column->get_nested_column();
+ }
+ column_ptr->insert_range_from(*src_column.column, 0, src_column.column->size());
+ }
+ } else {
+ columns[cur_col]->insert_range_from(*src_column.column, 0, src_column.column->size());
+ }
+ cur_col++;
+ }
+
+ // Fill grouping ID to tuple
+ for (auto slot_idx = 0; slot_idx < _grouping_list.size(); slot_idx++) {
+ DCHECK_LT(slot_idx, _virtual_tuple_desc->slots().size());
+ const SlotDescriptor* _virtual_slot_desc = _virtual_tuple_desc->slots()[slot_idx];
+ DCHECK_EQ(_virtual_slot_desc->type().type, _output_slots[cur_col]->type().type);
+ DCHECK_EQ(_virtual_slot_desc->col_name(), _output_slots[cur_col]->col_name());
+ int64_t val = _grouping_list[slot_idx][repeat_id_idx];
+ auto* column_ptr = columns[cur_col].get();
+ if (_output_slots[cur_col]->is_nullable()) {
+ auto* nullable_column = reinterpret_cast<ColumnNullable *>(columns[cur_col].get());
+ auto& null_map = nullable_column->get_null_map_data();
+ column_ptr = &nullable_column->get_nested_column();
+
+ for (size_t i = 0; i < child_block->rows(); ++i) {
+ null_map.push_back(0);
+ }
+ }
+
+ auto* col = assert_cast<ColumnVector<Int64> *>(column_ptr);
+ for (size_t i = 0; i < child_block->rows(); ++i) {
+ col->insert_value(val);
+ }
+ cur_col ++;
+ }
+
+ DCHECK_EQ(cur_col, column_size);
+
+ if (!columns.empty() && !columns[0]->empty()) {
+ auto n_columns = 0;
+ if (!mem_reuse) {
+ for (const auto slot_desc : _output_slots) {
+ output_block->insert(
+ ColumnWithTypeAndName(std::move(columns[n_columns++]), slot_desc->get_data_type_ptr(), slot_desc->col_name()));
+ }
+ } else {
+ columns.clear();
+ }
+ }
+ return Status::OK();
+}
+
+Status VRepeatNode::get_next(RuntimeState* state, Block* block, bool* eos) {
+ VLOG_CRITICAL << "VRepeatNode::get_next";
+ SCOPED_TIMER(_runtime_profile->total_time_counter());
+
+ if (state == NULL || block == NULL || eos == NULL)
+ return Status::InternalError("input is NULL pointer");
+
+ RETURN_IF_ERROR(exec_debug_action(TExecNodePhase::GETNEXT));
+ RETURN_IF_CANCELLED(state);
+ DCHECK(_repeat_id_idx >= 0);
+ for (const std::vector<int64_t>& v : _grouping_list) {
+ DCHECK(_repeat_id_idx <= (int)v.size());
+ }
+ DCHECK(block->rows() == 0);
+
+ // current child block has finished its repeat, get child's next block
+ if (_child_block->rows() == 0) {
+ if (_child_eos) {
+ *eos = true;
+ return Status::OK();
+ }
+
+ RETURN_IF_ERROR(child(0)->get_next(state, _child_block.get(), &_child_eos));
+
+ if (_child_block->rows() == 0) {
+ *eos = true;
+ return Status::OK();
+ }
+ }
+
+ RETURN_IF_ERROR(get_repeated_block(_child_block.get(), _repeat_id_idx, block));
+
+ _repeat_id_idx++;
+
+ int size = _repeat_id_list.size();
+ if (_repeat_id_idx >= size) {
+ release_block_memory(*_child_block.get());
+ _repeat_id_idx = 0;
+ }
+
+ _num_rows_returned += block->rows();
+ COUNTER_SET(_rows_returned_counter, _num_rows_returned);
+ VLOG_ROW << "VRepeatNode output rows: " << block->rows();
+ return Status::OK();
+}
+
+Status VRepeatNode::close(RuntimeState* state) {
+ VLOG_CRITICAL << "VRepeatNode::close";
+ if (is_closed()) {
+ return Status::OK();
+ }
+ release_block_memory(*_child_block.get());
+ RETURN_IF_ERROR(child(0)->close(state));
+ return ExecNode::close(state);
+}
+
+void VRepeatNode::debug_string(int indentation_level, std::stringstream* out) const {
+ *out << string(indentation_level * 2, ' ');
+ *out << "VRepeatNode(";
+ *out << "repeat pattern: [" << JoinElements(_repeat_id_list, ",") << "]\n";
+ *out << "add " << _grouping_list.size() << " columns. \n";
+ *out << "added column values: ";
+ for (const std::vector<int64_t>& v : _grouping_list) {
+ *out << "[" << JoinElements(v, ",") << "] ";
+ }
+ *out << "\n";
+ ExecNode::debug_string(indentation_level, out);
+ *out << ")";
+}
+} // namespace doris::vectorized
diff --git a/be/src/vec/exec/vrepeat_node.h b/be/src/vec/exec/vrepeat_node.h
new file mode 100644
index 0000000..cccf7ad
--- /dev/null
+++ b/be/src/vec/exec/vrepeat_node.h
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "exec/repeat_node.h"
+
+namespace doris {
+
+class ObjectPool;
+class TPlanNode;
+class DescriptorTbl;
+class RuntimeState;
+class Status;
+
+namespace vectorized {
+class VRepeatNode : public RepeatNode {
+public:
+ VRepeatNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs);
+ ~VRepeatNode() override = default;
+
+ virtual Status prepare(RuntimeState* state) override;
+ virtual Status open(RuntimeState* state) override;
+ virtual Status get_next(RuntimeState* state, Block* block, bool* eos) override;
+ virtual Status close(RuntimeState* state) override;
+
+protected:
+ virtual void debug_string(int indentation_level, std::stringstream* out) const override;
+
+private:
+ Status get_repeated_block(Block* child_block, int repeat_id_idx, Block* output_block);
+
+ std::unique_ptr<Block> _child_block;
+ std::vector<SlotDescriptor*> _child_slots;
+ std::vector<SlotDescriptor*> _output_slots;
+
+ // _virtual_tuple_id id used for GROUPING_ID().
+ TupleId _virtual_tuple_id;
+ const TupleDescriptor* _virtual_tuple_desc;
+};
+} // namespace vectorized
+} // namespace doris
diff --git a/be/src/vec/functions/function_grouping.cpp b/be/src/vec/functions/function_grouping.cpp
new file mode 100644
index 0000000..763dec2
--- /dev/null
+++ b/be/src/vec/functions/function_grouping.cpp
@@ -0,0 +1,25 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "function_grouping.h"
+
+namespace doris::vectorized {
+void register_function_grouping(SimpleFunctionFactory& factory) {
+ factory.register_function<FunctionGrouping>();
+ factory.register_function<FunctionGroupingId>();
+}
+}
diff --git a/be/src/vec/functions/function_grouping.h b/be/src/vec/functions/function_grouping.h
new file mode 100644
index 0000000..6bdaa7e
--- /dev/null
+++ b/be/src/vec/functions/function_grouping.h
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_FUNCTION_GROUPING_H
+#define DORIS_FUNCTION_GROUPING_H
+
+#include "vec/functions/simple_function_factory.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/functions/function_helpers.h"
+#include "vec/utils/util.hpp"
+#include "vec/data_types/get_least_supertype.h"
+
+namespace doris::vectorized {
+
+class FunctionGroupingBase : public IFunction {
+public:
+ size_t get_number_of_arguments() const override { return 1; }
+
+ bool use_default_implementation_for_constants() const override { return false; }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ DataTypePtr get_return_type_impl(const ColumnsWithTypeAndName& arguments) const override {
+ return std::make_shared<DataTypeInt64>();
+ }
+};
+
+class FunctionGrouping : public FunctionGroupingBase {
+public:
+ static constexpr auto name = "grouping";
+
+ static FunctionPtr create() { return std::make_shared<FunctionGrouping>(); }
+
+ String get_name() const override { return name; }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ const ColumnWithTypeAndName& src_column = block.get_by_position(arguments[0]);
+ const ColumnWithTypeAndName& rel_column = block.get_by_position(result);
+ if (!src_column.column)
+ return Status::InternalError("Illegal column " + src_column.column->get_name() + " of first argument of function " + name);
+
+ DCHECK(src_column.type->is_nullable() == true);
+ MutableColumnPtr res_column = rel_column.type->create_column();
+ auto* src_nullable_column = reinterpret_cast<ColumnNullable *>(const_cast<IColumn *>(src_column.column.get()));
+ res_column->insert_range_from(*src_nullable_column->get_nested_column_ptr().get(), 0, src_column.column->size());
+ block.get_by_position(result).column = std::move(res_column);
+ return Status::OK();
+ }
+};
+
+class FunctionGroupingId : public FunctionGroupingBase {
+public:
+ static constexpr auto name = "grouping_id";
+
+ static FunctionPtr create() { return std::make_shared<FunctionGroupingId>(); }
+
+ String get_name() const override { return name; }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ const ColumnWithTypeAndName& src_column = block.get_by_position(arguments[0]);
+ const ColumnWithTypeAndName& rel_column = block.get_by_position(result);
+ if (!src_column.column)
+ return Status::InternalError("Illegal column " + src_column.column->get_name() + " of first argument of function " + name);
+
+ DCHECK(src_column.type->is_nullable() == true);
+ MutableColumnPtr res_column = rel_column.type->create_column();
+ auto* src_nullable_column = reinterpret_cast<ColumnNullable *>(const_cast<IColumn *>(src_column.column.get()));
+ res_column->insert_range_from(*src_nullable_column->get_nested_column_ptr().get(), 0, src_column.column->size());
+ block.get_by_position(result).column = std::move(res_column);
+ return Status::OK();
+ }
+};
+}
+#endif //DORIS_FUNCTION_GROUPING_H
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index 0718930..45420cb 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -66,6 +66,7 @@ void register_function_like(SimpleFunctionFactory& factory);
void register_function_regexp(SimpleFunctionFactory& factory);
void register_function_random(SimpleFunctionFactory& factory);
void register_function_coalesce(SimpleFunctionFactory& factory);
++void register_function_grouping(SimpleFunctionFactory& factory);
class SimpleFunctionFactory {
using Creator = std::function<FunctionBuilderPtr()>;
@@ -181,6 +182,7 @@ public:
register_function_regexp(instance);
register_function_random(instance);
register_function_coalesce(instance);
+ register_function_grouping(instance);
});
return instance;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
index f62f606..e76253a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
@@ -1070,6 +1070,16 @@ public class SingleNodePlanner {
&& groupingInfo != null);
root = new RepeatNode(ctx_.getNextNodeId(), root, groupingInfo, groupByClause);
root.init(analyzer);
+ // set agg outtuple nullable
+ AggregateInfo aggInfo = selectStmt.getAggInfo();
+ TupleId aggOutTupleId = aggInfo.getOutputTupleId();
+ TupleDescriptor aggOutTupleDescriptor = analyzer.getDescTbl().getTupleDesc(aggOutTupleId);
+ int aggregateExprStartIndex = groupByClause.getGroupingExprs().size();
+ for (int i = 0; i < aggregateExprStartIndex; ++i) {
+ SlotDescriptor slot = aggOutTupleDescriptor.getSlots().get(i);
+ if (!slot.getIsNullable())
+ slot.setIsNullable(true);
+ }
return root;
}
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 8d3b6d9..ae7dc11 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -1397,10 +1397,10 @@ visible_functions = [
# grouping sets functions
[['grouping_id'], 'BIGINT', ['BIGINT'],
'_ZN5doris21GroupingSetsFunctions11grouping_idEPN9doris_udf15FunctionContextERKNS1_9BigIntValE',
- '', '', '', 'ALWAYS_NOT_NULLABLE'],
+ '', '', 'vec', 'ALWAYS_NOT_NULLABLE'],
[['grouping'], 'BIGINT', ['BIGINT'],
'_ZN5doris21GroupingSetsFunctions8groupingEPN9doris_udf15FunctionContextERKNS1_9BigIntValE',
- '' ,'', '', 'ALWAYS_NOT_NULLABLE'],
+ '' ,'', 'vec', 'ALWAYS_NOT_NULLABLE'],
]
# Except the following functions, other function will directly return
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 30/33: [Vectorized][Improvement] Speed up column filtering via SIMD (#7775)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 0a68fc3138057d536cae16e0c4052469cdd2c76e
Author: Zeno Yang <co...@qq.com>
AuthorDate: Mon Jan 17 16:54:40 2022 +0800
[Vectorized][Improvement] Speed up column filtering via SIMD (#7775)
---
be/src/vec/columns/column_decimal.cpp | 25 +++++++++++++++++++++++++
be/src/vec/columns/column_vector.cpp | 28 ++++++++++------------------
be/src/vec/columns/columns_common.cpp | 20 +++++++++-----------
be/src/vec/columns/columns_common.h | 30 ++++++++++++++++++++++++++++++
4 files changed, 74 insertions(+), 29 deletions(-)
diff --git a/be/src/vec/columns/column_decimal.cpp b/be/src/vec/columns/column_decimal.cpp
index 8e5ae12..5cc5853 100644
--- a/be/src/vec/columns/column_decimal.cpp
+++ b/be/src/vec/columns/column_decimal.cpp
@@ -162,6 +162,31 @@ ColumnPtr ColumnDecimal<T>::filter(const IColumn::Filter& filt, ssize_t result_s
const UInt8* filt_end = filt_pos + size;
const T* data_pos = data.data();
+ /** A slightly more optimized version.
+ * Based on the assumption that often pieces of consecutive values
+ * completely pass or do not pass the filter.
+ * Therefore, we will optimistically check the parts of `SIMD_BYTES` values.
+ */
+ static constexpr size_t SIMD_BYTES = 32;
+ const UInt8* filt_end_sse = filt_pos + size / SIMD_BYTES * SIMD_BYTES;
+
+ while (filt_pos < filt_end_sse) {
+ uint32_t mask = bytes32_mask_to_bits32_mask(filt_pos);
+
+ if (0xFFFFFFFF == mask) {
+ res_data.insert(data_pos, data_pos + SIMD_BYTES);
+ } else {
+ while (mask) {
+ const size_t idx = __builtin_ctzll(mask);
+ res_data.push_back(data_pos[idx]);
+ mask = mask & (mask - 1);
+ }
+ }
+
+ filt_pos += SIMD_BYTES;
+ data_pos += SIMD_BYTES;
+ }
+
while (filt_pos < filt_end) {
if (*filt_pos) res_data.push_back(*data_pos);
diff --git a/be/src/vec/columns/column_vector.cpp b/be/src/vec/columns/column_vector.cpp
index f6627d0..017ae29 100644
--- a/be/src/vec/columns/column_vector.cpp
+++ b/be/src/vec/columns/column_vector.cpp
@@ -26,6 +26,8 @@
#include <cmath>
#include <cstring>
+#include "runtime/datetime_value.h"
+#include "vec/columns/columns_common.h"
#include "vec/common/arena.h"
#include "vec/common/bit_cast.h"
#include "vec/common/exception.h"
@@ -33,12 +35,6 @@
#include "vec/common/sip_hash.h"
#include "vec/common/unaligned.h"
-#include "runtime/datetime_value.h"
-
-#ifdef __SSE2__
-#include <emmintrin.h>
-#endif
-
namespace doris::vectorized {
template <typename T>
@@ -237,34 +233,30 @@ ColumnPtr ColumnVector<T>::filter(const IColumn::Filter& filt, ssize_t result_si
const UInt8* filt_end = filt_pos + size;
const T* data_pos = data.data();
-#ifdef __SSE2__
/** A slightly more optimized version.
* Based on the assumption that often pieces of consecutive values
* completely pass or do not pass the filter.
* Therefore, we will optimistically check the parts of `SIMD_BYTES` values.
*/
-
- static constexpr size_t SIMD_BYTES = 16;
- const __m128i zero16 = _mm_setzero_si128();
+ static constexpr size_t SIMD_BYTES = 32;
const UInt8* filt_end_sse = filt_pos + size / SIMD_BYTES * SIMD_BYTES;
while (filt_pos < filt_end_sse) {
- int mask = _mm_movemask_epi8(_mm_cmpgt_epi8(
- _mm_loadu_si128(reinterpret_cast<const __m128i*>(filt_pos)), zero16));
+ uint32_t mask = bytes32_mask_to_bits32_mask(filt_pos);
- if (0 == mask) {
- /// Nothing is inserted.
- } else if (0xFFFF == mask) {
+ if (0xFFFFFFFF == mask) {
res_data.insert(data_pos, data_pos + SIMD_BYTES);
} else {
- for (size_t i = 0; i < SIMD_BYTES; ++i)
- if (filt_pos[i]) res_data.push_back(data_pos[i]);
+ while (mask) {
+ const size_t idx = __builtin_ctzll(mask);
+ res_data.push_back(data_pos[idx]);
+ mask = mask & (mask - 1);
+ }
}
filt_pos += SIMD_BYTES;
data_pos += SIMD_BYTES;
}
-#endif
while (filt_pos < filt_end) {
if (*filt_pos) res_data.push_back(*data_pos);
diff --git a/be/src/vec/columns/columns_common.cpp b/be/src/vec/columns/columns_common.cpp
index 02d650a..3045c5b 100644
--- a/be/src/vec/columns/columns_common.cpp
+++ b/be/src/vec/columns/columns_common.cpp
@@ -24,6 +24,7 @@
#include "vec/columns/column.h"
#include "vec/columns/column_vector.h"
+#include "vec/columns/columns_common.h"
#include "vec/common/typeid_cast.h"
namespace doris::vectorized {
@@ -173,18 +174,13 @@ void filter_arrays_impl_generic(const PaddedPODArray<T>& src_elems,
memcpy(&res_elems[elems_size_old], &src_elems[arr_offset], arr_size * sizeof(T));
};
-#ifdef __SSE2__
- const __m128i zero_vec = _mm_setzero_si128();
- static constexpr size_t SIMD_BYTES = 16;
+ static constexpr size_t SIMD_BYTES = 32;
const auto filt_end_aligned = filt_pos + size / SIMD_BYTES * SIMD_BYTES;
while (filt_pos < filt_end_aligned) {
- const auto mask = _mm_movemask_epi8(_mm_cmpgt_epi8(
- _mm_loadu_si128(reinterpret_cast<const __m128i*>(filt_pos)), zero_vec));
+ auto mask = bytes32_mask_to_bits32_mask(filt_pos);
- if (mask == 0) {
- /// SIMD_BYTES consecutive rows do not pass the filter
- } else if (mask == 0xffff) {
+ if (mask == 0xffffffff) {
/// SIMD_BYTES consecutive rows pass the filter
const auto first = offsets_pos == offsets_begin;
@@ -199,14 +195,16 @@ void filter_arrays_impl_generic(const PaddedPODArray<T>& src_elems,
res_elems.resize(elems_size_old + chunk_size);
memcpy(&res_elems[elems_size_old], &src_elems[chunk_offset], chunk_size * sizeof(T));
} else {
- for (size_t i = 0; i < SIMD_BYTES; ++i)
- if (filt_pos[i]) copy_array(offsets_pos + i);
+ while (mask) {
+ const size_t bit_pos = __builtin_ctzll(mask);
+ copy_array(offsets_pos + bit_pos);
+ mask = mask & (mask - 1);
+ }
}
filt_pos += SIMD_BYTES;
offsets_pos += SIMD_BYTES;
}
-#endif
while (filt_pos < filt_end) {
if (*filt_pos) copy_array(offsets_pos);
diff --git a/be/src/vec/columns/columns_common.h b/be/src/vec/columns/columns_common.h
index ef9c00c..83c38c4 100644
--- a/be/src/vec/columns/columns_common.h
+++ b/be/src/vec/columns/columns_common.h
@@ -22,10 +22,40 @@
#include "vec/columns/column.h"
+#ifdef __AVX2__
+#include <immintrin.h>
+#elif __SSE2__
+#include <emmintrin.h>
+#endif
+
/// Common helper methods for implementation of different columns.
namespace doris::vectorized {
+/// Transform 32-byte mask to 32-bit mask
+inline uint32_t bytes32_mask_to_bits32_mask(const uint8_t* filt_pos) {
+#ifdef __AVX2__
+ auto zero32 = _mm256_setzero_si256();
+ uint32_t mask = static_cast<uint32_t>(_mm256_movemask_epi8(_mm256_cmpgt_epi8(
+ _mm256_loadu_si256(reinterpret_cast<const __m256i*>(filt_pos)), zero32)));
+#elif __SSE2__
+ auto zero16 = _mm_setzero_si128();
+ uint32_t mask =
+ (static_cast<uint32_t>(_mm_movemask_epi8(_mm_cmpgt_epi8(
+ _mm_loadu_si128(reinterpret_cast<const __m128i*>(filt_pos)), zero16)))) |
+ ((static_cast<uint32_t>(_mm_movemask_epi8(_mm_cmpgt_epi8(
+ _mm_loadu_si128(reinterpret_cast<const __m128i*>(filt_pos + 16)), zero16)))
+ << 16) &
+ 0xffff0000);
+#else
+ uint32_t mask = 0;
+ for (size_t i = 0; i < 32; ++i) {
+ mask |= static_cast<uint32_t>(1 == *(filt_pos + i)) << i;
+ }
+#endif
+ return mask;
+}
+
/// Counts how many bytes of `filt` are greater than zero.
size_t count_bytes_in_filter(const IColumn::Filter& filt);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 10/33: [Function][Vec] add function coalesce (#7632)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit ebdc0bac985b488fda6f1a29d1c09ceedfe5315a
Author: zhangstar333 <87...@users.noreply.github.com>
AuthorDate: Thu Jan 6 19:33:25 2022 +0800
[Function][Vec] add function coalesce (#7632)
---
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/functions/function_coalesce.cpp | 143 +++++++++++++++++++++
be/src/vec/functions/simple_function_factory.h | 2 +
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js | 1 +
.../sql-functions/string-functions/coalesce.md | 62 +++++++++
.../sql-functions/string-functions/coalesce.md | 63 +++++++++
gensrc/script/doris_builtins_functions.py | 28 ++--
8 files changed, 287 insertions(+), 14 deletions(-)
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 71efde5..01c69eb 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -133,6 +133,7 @@ set(VEC_FILES
functions/function_ifnull.cpp
functions/nullif.cpp
functions/random.cpp
+ functions/function_coalesce.cpp
functions/function_date_or_datetime_computation.cpp
functions/function_date_or_datetime_to_string.cpp
functions/function_datetime_string_to_string.cpp
diff --git a/be/src/vec/functions/function_coalesce.cpp b/be/src/vec/functions/function_coalesce.cpp
new file mode 100644
index 0000000..65d544c
--- /dev/null
+++ b/be/src/vec/functions/function_coalesce.cpp
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "udf/udf.h"
+#include "vec/data_types/data_type_nothing.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/get_least_supertype.h"
+#include "vec/functions/function_helpers.h"
+#include "vec/functions/simple_function_factory.h"
+#include "vec/utils/util.hpp"
+
+namespace doris::vectorized {
+class FunctionCoalesce : public IFunction {
+public:
+ static constexpr auto name = "coalesce";
+
+ static FunctionPtr create() { return std::make_shared<FunctionCoalesce>(); }
+
+ String get_name() const override { return name; }
+
+ bool use_default_implementation_for_constants() const override { return false; }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ bool is_variadic() const override { return true; }
+
+ size_t get_number_of_arguments() const override { return 0; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ for (const auto& arg : arguments) {
+ if (!arg->is_nullable()) {
+ return arg;
+ }
+ }
+ return arguments[0];
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ DCHECK_GE(arguments.size(), 1);
+ ColumnNumbers filtered_args;
+ filtered_args.reserve(arguments.size());
+ for (const auto& arg : arguments) {
+ const auto& type = block.get_by_position(arg).type;
+ if (type->only_null()) {
+ continue;
+ }
+ filtered_args.push_back(arg);
+ if (!type->is_nullable()) {
+ break;
+ }
+ }
+
+ size_t remaining_rows = input_rows_count;
+ size_t argument_size = filtered_args.size();
+ std::vector<int> record_idx(input_rows_count, -1); //used to save column idx
+ MutableColumnPtr result_column;
+
+ DataTypePtr type = block.get_by_position(result).type;
+ if (!type->is_nullable()) {
+ result_column = type->create_column();
+ } else {
+ result_column = remove_nullable(type)->create_column();
+ }
+
+ result_column->reserve(input_rows_count);
+ auto return_type = std::make_shared<DataTypeUInt8>();
+ auto null_map = ColumnUInt8::create(input_rows_count, 1);
+ auto& null_map_data = null_map->get_data();
+ ColumnPtr argument_columns[argument_size];
+
+ for (size_t i = 0; i < argument_size; ++i) {
+ block.get_by_position(filtered_args[i]).column =
+ block.get_by_position(filtered_args[i])
+ .column->convert_to_full_column_if_const();
+ argument_columns[i] = block.get_by_position(filtered_args[i]).column;
+ if (auto* nullable = check_and_get_column<const ColumnNullable>(*argument_columns[i])) {
+ argument_columns[i] = nullable->get_nested_column_ptr();
+ }
+ }
+
+ for (size_t i = 0; i < argument_size && remaining_rows; ++i) {
+ const ColumnsWithTypeAndName is_not_null_col {block.get_by_position(filtered_args[i])};
+ Block temporary_block(is_not_null_col);
+ temporary_block.insert(ColumnWithTypeAndName {nullptr, return_type, ""});
+ auto func_is_not_null = SimpleFunctionFactory::instance().get_function("is_not_null_pred", is_not_null_col, return_type);
+ func_is_not_null->execute(context, temporary_block, {0}, 1, input_rows_count);
+
+ auto res_column = std::move(*temporary_block.get_by_position(1).column->convert_to_full_column_if_const()).mutate();
+ auto& res_map = assert_cast<ColumnVector<UInt8>*>(res_column.get())->get_data();
+ auto* res = res_map.data();
+
+ //TODO: if need to imporve performance in the feature, here it's could SIMD
+ for (size_t j = 0; j < input_rows_count && remaining_rows; ++j) {
+ if (res[j] && null_map_data[j]) {
+ null_map_data[j] = 0;
+ remaining_rows--;
+ record_idx[j] = i;
+ }
+ }
+ }
+ //TODO: According to the record results, fill in result one by one,
+ //that's fill in result use different methods for different types,
+ //because now the string types does not support random position writing,
+ //But other type could, so could check one column, and fill the not null value in result column,
+ //and then next column, this method may result in higher CPU cache
+ for (int row = 0; row < input_rows_count; ++row) {
+ if (record_idx[row] == -1) {
+ result_column->insert_default();
+ } else {
+ result_column->insert_from(*argument_columns[record_idx[row]].get(), row);
+ }
+ }
+
+ if (type->is_nullable()) {
+ block.replace_by_position(result, ColumnNullable::create(std::move(result_column), std::move(null_map)));
+ } else {
+ block.replace_by_position(result, std::move(result_column));
+ }
+
+ return Status::OK();
+ }
+};
+
+void register_function_coalesce(SimpleFunctionFactory& factory) {
+ factory.register_function<FunctionCoalesce>();
+}
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index c016d35..1d064bb 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -64,6 +64,7 @@ void register_function_function_ifnull(SimpleFunctionFactory& factory);
void register_function_like(SimpleFunctionFactory& factory);
void register_function_regexp(SimpleFunctionFactory& factory);
void register_function_random(SimpleFunctionFactory& factory);
+void register_function_coalesce(SimpleFunctionFactory& factory);
class SimpleFunctionFactory {
using Creator = std::function<FunctionBuilderPtr()>;
using FunctionCreators = std::unordered_map<std::string, Creator>;
@@ -176,6 +177,7 @@ public:
register_function_like(instance);
register_function_regexp(instance);
register_function_random(instance);
+ register_function_coalesce(instance);
});
return instance;
}
diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index 4a12a28..4adaa79 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -330,6 +330,7 @@ module.exports = [
"ascii",
"bit_length",
"char_length",
+ "coalesce",
"concat",
"concat_ws",
"ends_with",
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index b47618a..fdd2e01 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -334,6 +334,7 @@ module.exports = [
"ascii",
"bit_length",
"char_length",
+ "coalesce",
"concat",
"concat_ws",
"ends_with",
diff --git a/docs/en/sql-reference/sql-functions/string-functions/coalesce.md b/docs/en/sql-reference/sql-functions/string-functions/coalesce.md
new file mode 100644
index 0000000..23fef73
--- /dev/null
+++ b/docs/en/sql-reference/sql-functions/string-functions/coalesce.md
@@ -0,0 +1,62 @@
+---
+{
+ "title": "coalesce",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# coalesce
+## Description
+### Syntax
+
+`VARCHAR coalesce(VARCHAR, ...)`
+`...`
+`INT coalesce(INT, ...)`
+
+` coalesce ` function will return the first not null value. If it's all value is null, return null
+
+## example
+
+```
+MySQL> select coalesce(1,null,2);
++----------------------+
+| coalesce(1, NULL, 2) |
++----------------------+
+| 1 |
++----------------------+
+
+MySQL> select coalesce(null,"asd",1);
++--------------------------+
+| coalesce(NULL, 'asd', 1) |
++--------------------------+
+| asd |
++--------------------------+
+
+MySQL> select coalesce(null,null,null);
++----------------------------+
+| coalesce(NULL, NULL, NULL) |
++----------------------------+
+| NULL |
++----------------------------+
+```
+## keyword
+coalesce
diff --git a/docs/zh-CN/sql-reference/sql-functions/string-functions/coalesce.md b/docs/zh-CN/sql-reference/sql-functions/string-functions/coalesce.md
new file mode 100644
index 0000000..0ea6c2a
--- /dev/null
+++ b/docs/zh-CN/sql-reference/sql-functions/string-functions/coalesce.md
@@ -0,0 +1,63 @@
+---
+{
+ "title": "coalesce",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# coalesce
+## description
+### Syntax
+
+`VARCHAR coalesce(VARCHAR, ...)`
+`...`
+`INT coalesce(INT, ...)`
+
+`coalesce`函数会返回第一个非NULL的值，若全部为NULL，则返回NULL
+
+## example
+
+```
+MySQL> select coalesce(1,null,2);
++----------------------+
+| coalesce(1, NULL, 2) |
++----------------------+
+| 1 |
++----------------------+
+
+MySQL> select coalesce(null,"asd",1);
++--------------------------+
+| coalesce(NULL, 'asd', 1) |
++--------------------------+
+| asd |
++--------------------------+
+
+MySQL> select coalesce(null,null,null);
++----------------------------+
+| coalesce(NULL, NULL, NULL) |
++----------------------------+
+| NULL |
++----------------------------+
+
+```
+## keyword
+coalesce
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 5027627..51ecc89 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -805,21 +805,21 @@ visible_functions = [
[['ifnull'], 'VARCHAR', ['VARCHAR', 'VARCHAR'], '', '', '', 'vec', 'CUSTOM'],
[['ifnull'], 'STRING', ['STRING', 'STRING'], '', '', '', 'vec', 'CUSTOM'],
- [['coalesce'], 'BOOLEAN', ['BOOLEAN', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'TINYINT', ['TINYINT', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'SMALLINT', ['SMALLINT', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'INT', ['INT', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'BIGINT', ['BIGINT', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'LARGEINT', ['LARGEINT', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'FLOAT', ['FLOAT', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'DOUBLE', ['DOUBLE', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'DATETIME', ['DATETIME', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'DATE', ['DATE', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'DECIMALV2', ['DECIMALV2', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'BITMAP', ['BITMAP', '...'], '', '', '', '', 'CUSTOM'],
+ [['coalesce'], 'BOOLEAN', ['BOOLEAN', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'TINYINT', ['TINYINT', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'SMALLINT', ['SMALLINT', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'INT', ['INT', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'BIGINT', ['BIGINT', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'LARGEINT', ['LARGEINT', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'FLOAT', ['FLOAT', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'DOUBLE', ['DOUBLE', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'DATETIME', ['DATETIME', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'DATE', ['DATE', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'DECIMALV2', ['DECIMALV2', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'BITMAP', ['BITMAP', '...'], '', '', '', 'vec', 'CUSTOM'],
# The priority of varchar should be lower than decimal in IS_SUPERTYPE_OF mode.
- [['coalesce'], 'VARCHAR', ['VARCHAR', '...'], '', '', '', '', 'CUSTOM'],
- [['coalesce'], 'STRING', ['STRING', '...'], '', '', '', '', 'CUSTOM'],
+ [['coalesce'], 'VARCHAR', ['VARCHAR', '...'], '', '', '', 'vec', 'CUSTOM'],
+ [['coalesce'], 'STRING', ['STRING', '...'], '', '', '', 'vec', 'CUSTOM'],
[['esquery'], 'BOOLEAN', ['VARCHAR', 'VARCHAR'],
'_ZN5doris11ESFunctions5matchEPN'
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 22/33: [Vectorized][Bug] Fix get wrong result when select random column && fix get wrong has_null_tag (#7728)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a9d9c02a2fefc16c0eccc572626fac8b58a64d70
Author: Pxl <95...@qq.com>
AuthorDate: Wed Jan 12 15:55:58 2022 +0800
[Vectorized][Bug] Fix get wrong result when select random column && fix get wrong has_null_tag (#7728)
---
be/src/vec/columns/column.h | 3 +++
be/src/vec/columns/column_nullable.h | 5 +++--
be/src/vec/core/block.cpp | 29 +++++++++++++++++------------
be/src/vec/olap/block_reader.cpp | 23 +++++++++++++++--------
be/src/vec/olap/block_reader.h | 4 ++--
5 files changed, 40 insertions(+), 24 deletions(-)
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index 88f9a3a..5e41028 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -336,6 +336,9 @@ public:
// true iff column has null element
virtual bool has_null() const { return false; }
+ // true iff column has null element [0,size)
+ virtual bool has_null(size_t size) const { return false; }
+
/// It's a special kind of column, that contain single value, but is not a ColumnConst.
virtual bool is_dummy() const { return false; }
diff --git a/be/src/vec/columns/column_nullable.h b/be/src/vec/columns/column_nullable.h
index 9641788..8f47777 100644
--- a/be/src/vec/columns/column_nullable.h
+++ b/be/src/vec/columns/column_nullable.h
@@ -176,8 +176,9 @@ public:
/// Check that size of null map equals to size of nested column.
void check_consistency() const;
- bool has_null() const override {
- size_t size = get_null_map_data().size();
+ bool has_null() const override { return has_null(get_null_map_data().size()); }
+
+ bool has_null(size_t size) const override {
const UInt8* null_pos = get_null_map_data().data();
const UInt8* null_pos_end = get_null_map_data().data() + size;
#ifdef __SSE2__
diff --git a/be/src/vec/core/block.cpp b/be/src/vec/core/block.cpp
index 2aff751..d200a46 100644
--- a/be/src/vec/core/block.cpp
+++ b/be/src/vec/core/block.cpp
@@ -21,18 +21,18 @@
#include "vec/core/block.h"
#include <fmt/format.h>
+#include <snappy.h>
+
#include <iomanip>
#include <iterator>
#include <memory>
-#include <snappy.h>
#include "common/status.h"
#include "gen_cpp/data.pb.h"
#include "runtime/descriptors.h"
+#include "runtime/row_batch.h"
#include "runtime/tuple.h"
#include "runtime/tuple_row.h"
-#include "runtime/row_batch.h"
-
#include "vec/columns/column_const.h"
#include "vec/columns/column_nullable.h"
#include "vec/columns/column_vector.h"
@@ -692,8 +692,10 @@ Status Block::filter_block(Block* block, int filter_column_id, int column_to_kee
if (auto* nullable_column = check_and_get_column<ColumnNullable>(*filter_column)) {
ColumnPtr nested_column = nullable_column->get_nested_column_ptr();
- MutableColumnPtr mutable_holder = nested_column->use_count() == 1 ?
- nested_column->assume_mutable() : nested_column->clone_resized(nested_column->size());
+ MutableColumnPtr mutable_holder =
+ nested_column->use_count() == 1
+ ? nested_column->assume_mutable()
+ : nested_column->clone_resized(nested_column->size());
ColumnUInt8* concrete_column = typeid_cast<ColumnUInt8*>(mutable_holder.get());
if (!concrete_column) {
@@ -769,8 +771,8 @@ void Block::serialize(RowBatch* output_batch, const RowDescriptor& row_desc) {
}
}
-doris::Tuple* Block::deep_copy_tuple(const doris::TupleDescriptor& desc, MemPool* pool,
- int row, int column_offset, bool padding_char) {
+doris::Tuple* Block::deep_copy_tuple(const doris::TupleDescriptor& desc, MemPool* pool, int row,
+ int column_offset, bool padding_char) {
auto dst = reinterpret_cast<doris::Tuple*>(pool->allocate(desc.byte_size()));
for (int i = 0; i < desc.slots().size(); ++i) {
@@ -787,8 +789,9 @@ doris::Tuple* Block::deep_copy_tuple(const doris::TupleDescriptor& desc, MemPool
if (!slot_desc->type().is_string_type() && !slot_desc->type().is_date_type()) {
memcpy((void*)dst->get_slot(slot_desc->tuple_offset()), data_ref.data, data_ref.size);
- } else if (slot_desc->type().is_string_type() && slot_desc->type() != TYPE_OBJECT){
- memcpy((void*)dst->get_slot(slot_desc->tuple_offset()), (const void*)(&data_ref), sizeof(data_ref));
+ } else if (slot_desc->type().is_string_type() && slot_desc->type() != TYPE_OBJECT) {
+ memcpy((void*)dst->get_slot(slot_desc->tuple_offset()), (const void*)(&data_ref),
+ sizeof(data_ref));
// Copy the content of string
if (padding_char && slot_desc->type() == TYPE_CHAR) {
// serialize the content of string
@@ -800,7 +803,8 @@ doris::Tuple* Block::deep_copy_tuple(const doris::TupleDescriptor& desc, MemPool
} else {
auto str_ptr = pool->allocate(data_ref.size);
memcpy(str_ptr, data_ref.data, data_ref.size);
- dst->get_string_slot(slot_desc->tuple_offset())->ptr = reinterpret_cast<char *>(str_ptr);
+ dst->get_string_slot(slot_desc->tuple_offset())->ptr =
+ reinterpret_cast<char*>(str_ptr);
}
} else if (slot_desc->type() == TYPE_OBJECT) {
auto bitmap_value = (BitmapValue*)(data_ref.data);
@@ -812,7 +816,8 @@ doris::Tuple* Block::deep_copy_tuple(const doris::TupleDescriptor& desc, MemPool
bitmap_value->write(string_slot->ptr);
string_slot->len = size;
} else {
- VecDateTimeValue ts = *reinterpret_cast<const doris::vectorized::VecDateTimeValue*>(data_ref.data);
+ VecDateTimeValue ts =
+ *reinterpret_cast<const doris::vectorized::VecDateTimeValue*>(data_ref.data);
DateTimeValue dt;
ts.convert_vec_dt_to_dt(&dt);
memcpy((void*)dst->get_slot(slot_desc->tuple_offset()), &dt, sizeof(DateTimeValue));
@@ -903,7 +908,7 @@ std::unique_ptr<Block> Block::create_same_struct_block(size_t size) const {
auto temp_block = std::make_unique<Block>();
for (const auto& d : data) {
auto column = d.type->create_column();
- column->insert_many_defaults(size);
+ column->resize(size);
temp_block->insert({std::move(column), d.type, d.name});
}
return temp_block;
diff --git a/be/src/vec/olap/block_reader.cpp b/be/src/vec/olap/block_reader.cpp
index f7a1388..ec891f9 100644
--- a/be/src/vec/olap/block_reader.cpp
+++ b/be/src/vec/olap/block_reader.cpp
@@ -71,7 +71,7 @@ OLAPStatus BlockReader::_init_collect_iter(const ReaderParams& read_params,
return OLAP_SUCCESS;
}
-void BlockReader::_init_agg_state() {
+void BlockReader::_init_agg_state(const ReaderParams& read_params) {
if (_eof) {
return;
}
@@ -84,7 +84,10 @@ void BlockReader::_init_agg_state() {
auto& tablet_schema = tablet()->tablet_schema();
for (auto idx : _agg_columns_idx) {
- FieldAggregationMethod agg_method = tablet_schema.column(idx).aggregation();
+ FieldAggregationMethod agg_method =
+ tablet_schema
+ .column(read_params.origin_return_columns->at(_return_columns_loc[idx]))
+ .aggregation();
std::string agg_name =
TabletColumn::get_string_by_aggregation_type(agg_method) + agg_reader_suffix;
std::transform(agg_name.begin(), agg_name.end(), agg_name.begin(),
@@ -157,7 +160,7 @@ OLAPStatus BlockReader::init(const ReaderParams& read_params) {
break;
case KeysType::AGG_KEYS:
_next_block_func = &BlockReader::_agg_key_next_block;
- _init_agg_state();
+ _init_agg_state(read_params);
break;
default:
DCHECK(false) << "No next row function for type:" << tablet()->keys_type();
@@ -282,11 +285,11 @@ void BlockReader::_append_agg_data(MutableColumns& columns) {
void BlockReader::_update_agg_data(MutableColumns& columns) {
// copy data to stored block
- _copy_agg_data();
+ size_t copy_size = _copy_agg_data();
// calculate has_null_tag
for (auto idx : _agg_columns_idx) {
- _stored_has_null_tag[idx] = _stored_data_columns[idx]->has_null();
+ _stored_has_null_tag[idx] = _stored_data_columns[idx]->has_null(copy_size);
}
// calculate aggregate and insert
@@ -305,8 +308,10 @@ void BlockReader::_update_agg_data(MutableColumns& columns) {
_agg_data_counters.clear();
}
-void BlockReader::_copy_agg_data() {
- for (int i = 0; i < _stored_row_ref.size(); i++) {
+size_t BlockReader::_copy_agg_data() {
+ size_t copy_size = _stored_row_ref.size();
+
+ for (size_t i = 0; i < copy_size; i++) {
auto& ref = _stored_row_ref[i];
_temp_ref_map[ref.block].emplace_back(ref.row_pos, i);
}
@@ -315,7 +320,7 @@ void BlockReader::_copy_agg_data() {
auto& dst_column = _stored_data_columns[idx];
if (_stored_has_string_tag[idx]) {
//string type should replace ordered
- for (int i = 0; i < _stored_row_ref.size(); i++) {
+ for (size_t i = 0; i < copy_size; i++) {
auto& ref = _stored_row_ref[i];
dst_column->replace_column_data(*ref.block->get_by_position(idx).column,
ref.row_pos, i);
@@ -334,6 +339,8 @@ void BlockReader::_copy_agg_data() {
it.second.clear();
}
_stored_row_ref.clear();
+
+ return copy_size;
}
void BlockReader::_update_agg_value(MutableColumns& columns, int begin, int end, bool is_close) {
diff --git a/be/src/vec/olap/block_reader.h b/be/src/vec/olap/block_reader.h
index 02cb570..0fba0be 100644
--- a/be/src/vec/olap/block_reader.h
+++ b/be/src/vec/olap/block_reader.h
@@ -73,7 +73,7 @@ private:
OLAPStatus _init_collect_iter(const ReaderParams& read_params,
std::vector<RowsetReaderSharedPtr>* valid_rs_readers);
- void _init_agg_state();
+ void _init_agg_state(const ReaderParams& read_params);
void _insert_data_normal(MutableColumns& columns);
@@ -81,7 +81,7 @@ private:
void _update_agg_data(MutableColumns& columns);
- void _copy_agg_data();
+ size_t _copy_agg_data();
void _update_agg_value(MutableColumns& columns, int begin, int end, bool is_close = true);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 08/33: [Bug] Fix bug of cast expr nullable and ifnull function (#7626)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 24e1d64a6b26be2f97477d39d87caed8ee0f5f8e
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Jan 5 23:56:22 2022 -0600
[Bug] Fix bug of cast expr nullable and ifnull function (#7626)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/runtime/descriptors.h | 1 -
be/src/runtime/fold_constant_executor.cpp | 8 ++--
be/src/vec/exec/join/vhash_join_node.cpp | 2 +-
be/src/vec/exec/vunion_node.cpp | 2 +
be/src/vec/functions/function.cpp | 2 +-
.../function_date_or_datetime_computation.h | 4 +-
be/src/vec/functions/function_ifnull.h | 43 ++++++++++++----------
be/src/vec/sink/vtabet_sink.cpp | 1 +
.../java/org/apache/doris/analysis/CastExpr.java | 7 ++++
.../apache/doris/analysis/FunctionCallExpr.java | 2 +-
.../apache/doris/rewrite/FoldConstantsRule.java | 9 ++++-
11 files changed, 51 insertions(+), 30 deletions(-)
diff --git a/be/src/runtime/descriptors.h b/be/src/runtime/descriptors.h
index 97f9712..ad43209 100644
--- a/be/src/runtime/descriptors.h
+++ b/be/src/runtime/descriptors.h
@@ -381,7 +381,6 @@ public:
int get_row_size() const;
int num_materialized_slots() const {
- DCHECK(_num_materialized_slots != 0);
return _num_materialized_slots;
}
diff --git a/be/src/runtime/fold_constant_executor.cpp b/be/src/runtime/fold_constant_executor.cpp
index 9781c2f..f093c04 100644
--- a/be/src/runtime/fold_constant_executor.cpp
+++ b/be/src/runtime/fold_constant_executor.cpp
@@ -127,9 +127,11 @@ Status FoldConstantExecutor::fold_constant_vexpr(
}
vectorized::Block tmp_block;
+ tmp_block.insert({vectorized::ColumnUInt8::create(1),
+ std::make_shared<vectorized::DataTypeUInt8>(), ""});
int result_column = -1;
// calc vexpr
- ctx->execute(&tmp_block, &result_column);
+ RETURN_IF_ERROR(ctx->execute(&tmp_block, &result_column));
DCHECK(result_column != -1);
PrimitiveType root_type = ctx->root()->type().type;
// covert to thrift type
@@ -139,7 +141,7 @@ Status FoldConstantExecutor::fold_constant_vexpr(
PExprResult expr_result;
string result;
const auto& column_ptr = tmp_block.get_by_position(result_column).column;
- if (column_ptr->is_nullable() && column_ptr->is_null_at(0)) {
+ if (column_ptr->is_null_at(0)) {
expr_result.set_success(false);
} else {
expr_result.set_success(true);
@@ -194,7 +196,7 @@ Status FoldConstantExecutor::_init(const TQueryGlobals& query_globals) {
template <typename Context>
Status FoldConstantExecutor::_prepare_and_open(Context* ctx) {
- ctx->prepare(_runtime_state.get(), RowDescriptor(), _mem_tracker);
+ RETURN_IF_ERROR(ctx->prepare(_runtime_state.get(), RowDescriptor(), _mem_tracker));
return ctx->open(_runtime_state.get());
}
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp b/be/src/vec/exec/join/vhash_join_node.cpp
index 7606783..4533cae 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -133,7 +133,7 @@ struct ProcessRuntimeFilterBuild {
RETURN_IF_ERROR(runtime_filter_slots->init(state, hash_table_ctx.hash_table.get_size()));
- if (!runtime_filter_slots->empty()) {
+ if (!runtime_filter_slots->empty() && !_join_node->_inserted_rows.empty()) {
{
SCOPED_TIMER(_join_node->_push_compute_timer);
runtime_filter_slots->insert(_join_node->_inserted_rows);
diff --git a/be/src/vec/exec/vunion_node.cpp b/be/src/vec/exec/vunion_node.cpp
index 122eafa..1fa4da4 100644
--- a/be/src/vec/exec/vunion_node.cpp
+++ b/be/src/vec/exec/vunion_node.cpp
@@ -181,6 +181,8 @@ Status VUnionNode::get_next_const(RuntimeState* state, Block* block) {
MutableBlock(Block(VectorizedUtils::create_columns_with_type_and_name(row_desc())));
for (; _const_expr_list_idx < _const_expr_lists.size(); ++_const_expr_list_idx) {
Block tmp_block;
+ tmp_block.insert({vectorized::ColumnUInt8::create(1),
+ std::make_shared<vectorized::DataTypeUInt8>(), ""});
int const_expr_lists_size = _const_expr_lists[_const_expr_list_idx].size();
std::vector<int> result_list(const_expr_lists_size);
for (size_t i = 0; i < const_expr_lists_size; ++i) {
diff --git a/be/src/vec/functions/function.cpp b/be/src/vec/functions/function.cpp
index d3a910e..9ed3edb 100644
--- a/be/src/vec/functions/function.cpp
+++ b/be/src/vec/functions/function.cpp
@@ -66,7 +66,7 @@ ColumnPtr wrap_in_nullable(const ColumnPtr& src, const Block& block, const Colum
null_map_column->clone_resized(null_map_column->size());
} else {
MutableColumnPtr mutable_result_null_map_column =
- (*std::move(result_null_map_column)).mutate();
+ (*std::move(result_null_map_column)).assume_mutable();
NullMap& result_null_map =
assert_cast<ColumnUInt8&>(*mutable_result_null_map_column).get_data();
diff --git a/be/src/vec/functions/function_date_or_datetime_computation.h b/be/src/vec/functions/function_date_or_datetime_computation.h
index 33c7588..607039c 100644
--- a/be/src/vec/functions/function_date_or_datetime_computation.h
+++ b/be/src/vec/functions/function_date_or_datetime_computation.h
@@ -434,11 +434,11 @@ struct CurrentDateImpl {
template<typename FunctionName>
struct CurrentTimeImpl {
- using ReturnType = DataTypeInt64;
+ using ReturnType = DataTypeFloat64;
static constexpr auto name = FunctionName::name;
static Status execute(FunctionContext* context, Block& block, size_t result,
size_t input_rows_count) {
- auto col_to = ColumnVector<Int64>::create();
+ auto col_to = ColumnVector<Float64>::create();
VecDateTimeValue dtv;
if (dtv.from_unixtime(context->impl()->state()->timestamp_ms() / 1000,
context->impl()->state()->timezone_obj())) {
diff --git a/be/src/vec/functions/function_ifnull.h b/be/src/vec/functions/function_ifnull.h
index b18de80..26fe18c 100644
--- a/be/src/vec/functions/function_ifnull.h
+++ b/be/src/vec/functions/function_ifnull.h
@@ -42,6 +42,9 @@ public:
bool use_default_implementation_for_constants() const override { return false; }
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ if (!arguments[0]->is_nullable() && arguments[1]->is_nullable()) {
+ return reinterpret_cast<const DataTypeNullable*>(arguments[1].get())->get_nested_type();
+ }
return arguments[1];
}
@@ -55,42 +58,42 @@ public:
block.get_by_position(result).column = block.get_by_position(arguments[1]).column;
return Status::OK();
}
- const ColumnWithTypeAndName new_result_column {
- block.get_by_position(result),
+
+ ColumnWithTypeAndName null_column_arg0 {
+ nullptr, std::make_shared<DataTypeUInt8>(),""
};
- const ColumnWithTypeAndName new_column {
- col_left,
+ ColumnWithTypeAndName nested_column_arg0 {
+ nullptr, col_left.type, ""
};
+
/// implement isnull(col_left) logic
if (auto* nullable = check_and_get_column<ColumnNullable>(*col_left.column)) {
- block.get_by_position(arguments[0]).column = nullable->get_null_map_column_ptr();
+ null_column_arg0.column = nullable->get_null_map_column_ptr();
+ nested_column_arg0.column = nullable->get_nested_column_ptr();
+ nested_column_arg0.type = reinterpret_cast<const DataTypeNullable*>(
+ nested_column_arg0.type.get())->get_nested_type();
} else {
block.get_by_position(result).column = col_left.column;
return Status::OK();
}
const ColumnsWithTypeAndName if_columns
{
- block.get_by_position(arguments[0]),
+ null_column_arg0,
block.get_by_position(arguments[1]),
- new_column,
+ nested_column_arg0
};
Block temporary_block(
- {block.get_by_position(arguments[0]),
- block.get_by_position(arguments[1]),
- new_column,
- new_result_column
+ {
+ null_column_arg0,
+ block.get_by_position(arguments[1]),
+ nested_column_arg0,
+ block.get_by_position(result),
});
- auto func_if = SimpleFunctionFactory::instance().get_function("if", if_columns, make_nullable(new_result_column.type));
+
+ auto func_if = SimpleFunctionFactory::instance().get_function("if", if_columns, block.get_by_position(result).type);
func_if->execute(context, temporary_block, {0, 1, 2}, 3, input_rows_count);
- /// need to handle nullable type and not nullable type differently,
- /// because `IF` function always return nullable type, but result type is not always
- if (block.get_by_position(result).type->is_nullable()) {
- block.get_by_position(result).column = temporary_block.get_by_position(3).column;
- } else {
- auto cols = check_and_get_column<ColumnNullable>(temporary_block.get_by_position(3).column.get());
- block.replace_by_position(result, std::move(cols->get_nested_column_ptr()));
- }
+ block.get_by_position(result).column = temporary_block.get_by_position(3).column;
return Status::OK();
}
};
diff --git a/be/src/vec/sink/vtabet_sink.cpp b/be/src/vec/sink/vtabet_sink.cpp
index 685f017..3b09cae 100644
--- a/be/src/vec/sink/vtabet_sink.cpp
+++ b/be/src/vec/sink/vtabet_sink.cpp
@@ -129,6 +129,7 @@ int VOlapTableSink::_validate_data(doris::RuntimeState* state, doris::vectorized
for (int i = 0; i < _output_tuple_desc->slots().size(); ++i) {
SlotDescriptor* desc = _output_tuple_desc->slots()[i];
+ block->get_by_position(i).column = block->get_by_position(i).column->convert_to_full_column_if_const();
const auto& column = block->get_by_position(i).column;
if (desc->is_nullable() && desc->type() == TYPE_OBJECT) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
index f071e0f..2a629f9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
@@ -451,4 +451,11 @@ public class CastExpr extends Expr {
}
return -1;
}
+
+ @Override
+ public boolean isNullable() {
+ return children.get(0).isNullable() ||
+ (children.get(0).getType().isStringType() && !getType().isStringType()) ||
+ (!children.get(0).getType().isDateType() && getType().isDateType());
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index f6a350a..6fa7c69 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -1053,7 +1053,7 @@ public class FunctionCallExpr extends Expr {
// TODO: we can't correctly determine const-ness before analyzing 'fn_'. We should
// rework logic so that we do not call this function on unanalyzed exprs.
// Aggregate functions are never constant.
- if (fn instanceof AggregateFunction) return false;
+ if (fn instanceof AggregateFunction || fn == null) return false;
final String fnName = this.fnName.getFunction();
// Non-deterministic functions are never constant.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
index 8eaac29..628740a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
@@ -27,6 +27,7 @@ import org.apache.doris.analysis.LiteralExpr;
import org.apache.doris.analysis.NullLiteral;
import org.apache.doris.analysis.SysVariableDesc;
import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Function;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
@@ -194,6 +195,12 @@ public class FoldConstantsRule implements ExprRewriteRule {
Analyzer analyzer, Map<String, Expr> sysVarMap, Map<String, Expr> infoFnMap)
throws AnalysisException {
if (expr.isConstant()) {
+ if (VectorizedUtil.isVectorized()) {
+ Function fn = expr.getFn();
+ if (fn != null && (fn.functionName().equalsIgnoreCase("curtime") ||
+ fn.functionName().equalsIgnoreCase("current_time")))
+ return;
+ }
// Do not constant fold cast(null as dataType) because we cannot preserve the
// cast-to-types and that can lead to query failures, e.g., CTAS
if (expr instanceof CastExpr) {
@@ -216,7 +223,7 @@ public class FoldConstantsRule implements ExprRewriteRule {
getInfoFnExpr(expr, infoFnMap);
return;
}
- constExprMap.put(expr.getId().toString(),expr.treeToThrift());
+ constExprMap.put(expr.getId().toString(), expr.treeToThrift());
oriConstMap.put(expr.getId().toString(), expr);
} else {
recursiveGetChildrenConstExpr(expr, constExprMap, oriConstMap, analyzer, sysVarMap, infoFnMap);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 11/33: [Bug] Fix function nulllable not match and largetint cast failed (#7659)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit da43e38a71ac7b5394e15d1d52ea055f50684f83
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Thu Jan 6 21:52:51 2022 -0600
[Bug] Fix function nulllable not match and largetint cast failed (#7659)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/vec/common/cow.h | 8 ++++----
be/src/vec/core/block.cpp | 6 ++++--
be/src/vec/data_types/data_type_number_base.cpp | 2 +-
be/src/vec/functions/date_time_transforms.h | 8 +++++---
be/src/vec/io/io_helper.h | 18 ++++--------------
gensrc/script/doris_builtins_functions.py | 10 +++++-----
6 files changed, 23 insertions(+), 29 deletions(-)
diff --git a/be/src/vec/common/cow.h b/be/src/vec/common/cow.h
index 08edb89..58ae14d 100644
--- a/be/src/vec/common/cow.h
+++ b/be/src/vec/common/cow.h
@@ -105,10 +105,6 @@ protected:
return *this;
}
- unsigned int use_count() const {
- return ref_counter.load();
- }
-
void add_ref() {
++ref_counter;
}
@@ -265,6 +261,10 @@ protected:
public:
using MutablePtr = mutable_ptr<Derived>;
+ unsigned int use_count() const {
+ return ref_counter.load();
+ }
+
protected:
template <typename T>
class immutable_ptr : public intrusive_ptr<const T> {
diff --git a/be/src/vec/core/block.cpp b/be/src/vec/core/block.cpp
index b52257d..a5e445d 100644
--- a/be/src/vec/core/block.cpp
+++ b/be/src/vec/core/block.cpp
@@ -646,7 +646,8 @@ void Block::clear_column_data(int column_size) noexcept {
}
}
for (auto& d : data) {
- (*std::move(d.column)).mutate()->clear();
+ DCHECK(d.column->use_count() == 1);
+ (*std::move(d.column)).assume_mutable()->clear();
}
}
@@ -691,7 +692,8 @@ Status Block::filter_block(Block* block, int filter_column_id, int column_to_kee
if (auto* nullable_column = check_and_get_column<ColumnNullable>(*filter_column)) {
ColumnPtr nested_column = nullable_column->get_nested_column_ptr();
- MutableColumnPtr mutable_holder = (*std::move(nested_column)).mutate();
+ MutableColumnPtr mutable_holder = nested_column->use_count() == 1 ?
+ nested_column->assume_mutable() : nested_column->clone_resized(nested_column->size());
ColumnUInt8* concrete_column = typeid_cast<ColumnUInt8*>(mutable_holder.get());
if (!concrete_column) {
diff --git a/be/src/vec/data_types/data_type_number_base.cpp b/be/src/vec/data_types/data_type_number_base.cpp
index ee94a37..01a4248 100644
--- a/be/src/vec/data_types/data_type_number_base.cpp
+++ b/be/src/vec/data_types/data_type_number_base.cpp
@@ -35,7 +35,7 @@ namespace doris::vectorized {
template <typename T>
void DataTypeNumberBase<T>::to_string(const IColumn& column, size_t row_num,
BufferWritable& ostr) const {
- if constexpr (std::is_same<T, __int128_t>::value || std::is_same<T, UInt128>::value) {
+ if constexpr (std::is_same<T, UInt128>::value) {
std::string hex = int128_to_string(
assert_cast<const ColumnVector<T>&>(*column.convert_to_full_column_if_const().get())
.get_data()[row_num]);
diff --git a/be/src/vec/functions/date_time_transforms.h b/be/src/vec/functions/date_time_transforms.h
index a34b53d..eaab918 100644
--- a/be/src/vec/functions/date_time_transforms.h
+++ b/be/src/vec/functions/date_time_transforms.h
@@ -56,6 +56,7 @@ TIME_FUNCTION_IMPL(WeekOfYearImpl, weekofyear, week(mysql_week_mode(3)));
TIME_FUNCTION_IMPL(DayOfYearImpl, dayofyear, day_of_year());
TIME_FUNCTION_IMPL(DayOfMonthImpl, dayofmonth, day());
TIME_FUNCTION_IMPL(DayOfWeekImpl, dayofweek, day_of_week());
+// TODO: the method should be always not nullable
TIME_FUNCTION_IMPL(ToDaysImpl, to_days, daynr());
TIME_FUNCTION_IMPL(ToYearWeekImpl, yearweek, year_week(mysql_week_mode(0)));
struct ToDateImpl {
@@ -92,7 +93,7 @@ struct DayNameImpl {
res_data[offset - 1] = 0;
} else {
auto len = strlen(day_name);
- memcpy_small_allow_read_write_overflow15(&res_data[offset], day_name, len);
+ memcpy(&res_data[offset], day_name, len);
offset += len + 1;
res_data[offset - 1] = 0;
}
@@ -113,8 +114,8 @@ struct MonthNameImpl {
res_data[offset - 1] = 0;
} else {
auto len = strlen(month_name);
- memcpy_small_allow_read_write_overflow15(&res_data[offset], month_name, len);
- offset += len + 1;
+ memcpy(&res_data[offset], month_name, len);
+ offset += (len + 1);
res_data[offset - 1] = 0;
}
return offset;
@@ -148,6 +149,7 @@ struct DateFormatImpl {
}
};
+// TODO: This function should be depend on argments not always nullable
struct FromUnixTimeImpl {
using FromType = Int32;
diff --git a/be/src/vec/io/io_helper.h b/be/src/vec/io/io_helper.h
index fb9371f..ab3caa7 100644
--- a/be/src/vec/io/io_helper.h
+++ b/be/src/vec/io/io_helper.h
@@ -54,19 +54,9 @@ inline Int128 decimal_scale_multiplier<Int128>(UInt32 scale) {
}
inline std::string int128_to_string(__int128_t value) {
- char buffer[128];
- char* d = std::end(buffer);
- do {
- --d;
- *d = "0123456789"[value % 10];
- value /= 10;
- } while (value != 0);
- if (value < 0) {
- --d;
- *d = '-';
- }
- int len = std::end(buffer) - d;
- return std::string(d, len);
+ fmt::memory_buffer buffer;
+ fmt::format_to(buffer, "{}", value);
+ return std::string(buffer.data(), buffer.size());
}
inline std::string int128_to_string(UInt128 value) {
@@ -84,7 +74,7 @@ void write_text(Decimal<T> value, UInt32 scale, std::ostream& ostr) {
if (scale) {
whole_part = value / decimal_scale_multiplier<T>(scale);
}
- if constexpr (std::is_same<T, __int128_t>::value || std::is_same<T, UInt128>::value) {
+ if constexpr (std::is_same<T, __int128_t>::value) {
ostr << int128_to_string(whole_part);
} else {
ostr << whole_part;
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 51ecc89..1399c97 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -139,14 +139,14 @@ visible_functions = [
'', '', 'vec', 'ALWAYS_NULLABLE'],
[['from_unixtime'], 'VARCHAR', ['INT'],
'_ZN5doris18TimestampFunctions9from_unixEPN9doris_udf15FunctionContextERKNS1_6IntValE',
- '', '', 'vec', ''],
+ '', '', 'vec', 'ALWAYS_NULLABLE'],
[['from_unixtime'], 'VARCHAR', ['INT', 'VARCHAR'],
'_ZN5doris18TimestampFunctions9from_unixEPN9doris_udf'
'15FunctionContextERKNS1_6IntValERKNS1_9StringValE',
'_ZN5doris18TimestampFunctions14format_prepareEPN9doris_udf'
'15FunctionContextENS2_18FunctionStateScopeE',
'_ZN5doris18TimestampFunctions12format_closeEPN9doris_udf'
- '15FunctionContextENS2_18FunctionStateScopeE', 'vec', ''],
+ '15FunctionContextENS2_18FunctionStateScopeE', 'vec', 'ALWAYS_NULLABLE'],
[['from_unixtime'], 'VARCHAR', ['INT', 'STRING'],
'_ZN5doris18TimestampFunctions9from_unixEPN9doris_udf'
'15FunctionContextERKNS1_6IntValERKNS1_9StringValE',
@@ -175,7 +175,7 @@ visible_functions = [
'', '', 'vec', 'ALWAYS_NULLABLE'],
[['to_days'], 'INT', ['DATE'],
'_ZN5doris18TimestampFunctions7to_daysEPN9doris_udf15FunctionContextERKNS1_11DateTimeValE',
- '', '', 'vec', ''],
+ '', '', 'vec', 'ALWAYS_NULLABLE'],
[['year'], 'INT', ['DATETIME'],
'_ZN5doris18TimestampFunctions4yearEPN9doris_udf15FunctionContextERKNS1_11DateTimeValE',
@@ -528,9 +528,9 @@ visible_functions = [
# Math builtin functions
[['pi'], 'DOUBLE', [],
- '_ZN5doris13MathFunctions2piEPN9doris_udf15FunctionContextE', '', '', 'vec', 'ALWAYS_NULLABLE'],
+ '_ZN5doris13MathFunctions2piEPN9doris_udf15FunctionContextE', '', '', 'vec', 'ALWAYS_NOT_NULLABLE'],
[['e'], 'DOUBLE', [],
- '_ZN5doris13MathFunctions1eEPN9doris_udf15FunctionContextE', '', '', 'vec', 'ALWAYS_NULLABLE'],
+ '_ZN5doris13MathFunctions1eEPN9doris_udf15FunctionContextE', '', '', 'vec', 'ALWAYS_NOT_NULLABLE'],
[['abs'], 'DOUBLE', ['DOUBLE'],
'_ZN5doris13MathFunctions3absEPN9doris_udf15FunctionContextERKNS1_9DoubleValE', '', '', 'vec', ''],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 28/33: [Vectorized][Improvement] Enhancement unit test for vectorized function (#7750)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit d1c75a05c4c6d0993a3bbb8c8657240e5a44dead
Author: Pxl <95...@qq.com>
AuthorDate: Mon Jan 17 11:18:02 2022 +0800
[Vectorized][Improvement] Enhancement unit test for vectorized function (#7750)
---
be/src/vec/core/types.h | 7 +
be/src/vec/data_types/data_type_decimal.h | 3 +-
be/src/vec/functions/function.h | 3 +-
be/test/vec/function/CMakeLists.txt | 1 -
be/test/vec/function/function_abs_test.cpp | 92 -----
be/test/vec/function/function_arithmetic_test.cpp | 49 +--
be/test/vec/function/function_bitmap_test.cpp | 39 +--
be/test/vec/function/function_comparison_test.cpp | 50 +--
be/test/vec/function/function_hash_test.cpp | 43 +--
be/test/vec/function/function_ifnull_test.cpp | 51 +--
be/test/vec/function/function_like_test.cpp | 130 ++++---
be/test/vec/function/function_math_test.cpp | 231 ++++++++----
be/test/vec/function/function_nullif_test.cpp | 39 +--
be/test/vec/function/function_string_test.cpp | 249 +++++++------
be/test/vec/function/function_test_util.h | 121 ++++++-
be/test/vec/function/function_time_test.cpp | 405 +++++++++-------------
run-be-ut.sh | 26 +-
17 files changed, 730 insertions(+), 809 deletions(-)
diff --git a/be/src/vec/core/types.h b/be/src/vec/core/types.h
index 807e79a..dddeff8 100644
--- a/be/src/vec/core/types.h
+++ b/be/src/vec/core/types.h
@@ -25,6 +25,7 @@
#include <string>
#include <vector>
+#include "util/binary_cast.hpp"
#include "util/bitmap_value.h"
namespace doris::vectorized {
@@ -238,6 +239,12 @@ struct Decimal {
Decimal(const T& value_) : value(value_) {}
+ static Decimal double_to_decimal(double value_) {
+ DecimalV2Value decimal_value;
+ decimal_value.assign_from_double(value_);
+ return Decimal(binary_cast<DecimalV2Value, T>(decimal_value));
+ }
+
template <typename U>
Decimal(const Decimal<U>& x) : value(x) {}
diff --git a/be/src/vec/data_types/data_type_decimal.h b/be/src/vec/data_types/data_type_decimal.h
index 5b6ab86..c99e507 100644
--- a/be/src/vec/data_types/data_type_decimal.h
+++ b/be/src/vec/data_types/data_type_decimal.h
@@ -95,7 +95,8 @@ public:
static constexpr size_t max_precision() { return max_decimal_precision<T>(); }
- DataTypeDecimal(UInt32 precision_, UInt32 scale_) : precision(precision_), scale(scale_) {
+ DataTypeDecimal(UInt32 precision_ = 27, UInt32 scale_ = 9)
+ : precision(precision_), scale(scale_) {
if (UNLIKELY(precision < 1 || precision > max_precision())) {
LOG(FATAL) << fmt::format("Precision {} is out of bounds", precision);
}
diff --git a/be/src/vec/functions/function.h b/be/src/vec/functions/function.h
index 9a43fe1..bc76a10 100644
--- a/be/src/vec/functions/function.h
+++ b/be/src/vec/functions/function.h
@@ -298,7 +298,8 @@ public:
is_date_or_datetime(get_return_type(arguments)->is_nullable()
? ((DataTypeNullable*)get_return_type(arguments).get())
->get_nested_type()
- : get_return_type(arguments))));
+ : get_return_type(arguments))))
+ << " with " << return_type->get_name() << " and " << func_return_type->get_name();
return build_impl(arguments, return_type);
}
diff --git a/be/test/vec/function/CMakeLists.txt b/be/test/vec/function/CMakeLists.txt
index e715604..6ae9b05 100644
--- a/be/test/vec/function/CMakeLists.txt
+++ b/be/test/vec/function/CMakeLists.txt
@@ -18,7 +18,6 @@
# where to put generated libraries
set(EXECUTABLE_OUTPUT_PATH "${BUILD_DIR}/test/vec/function")
-ADD_BE_TEST(function_abs_test)
ADD_BE_TEST(function_bitmap_test)
ADD_BE_TEST(function_comparison_test)
ADD_BE_TEST(function_hash_test)
diff --git a/be/test/vec/function/function_abs_test.cpp b/be/test/vec/function/function_abs_test.cpp
deleted file mode 100644
index e6bcdb0..0000000
--- a/be/test/vec/function/function_abs_test.cpp
+++ /dev/null
@@ -1,92 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include <gtest/gtest.h>
-
-#include <iostream>
-#include <string>
-
-#include "exec/schema_scanner.h"
-#include "runtime/row_batch.h"
-#include "runtime/tuple_row.h"
-#include "vec/functions/simple_function_factory.h"
-
-namespace doris {
-
-TEST(ABSTest, ABSTest) {
- SchemaScanner::ColumnDesc column_descs[] = {{"k1", TYPE_SMALLINT, sizeof(int16_t), false},
- {"k2", TYPE_INT, sizeof(int32_t), false},
- {"k3", TYPE_DOUBLE, sizeof(double), false}};
- SchemaScanner schema_scanner(column_descs, 3);
- ObjectPool object_pool;
- SchemaScannerParam param;
- schema_scanner.init(¶m, &object_pool);
-
- auto tuple_desc = const_cast<TupleDescriptor*>(schema_scanner.tuple_desc());
- RowDescriptor row_desc(tuple_desc, false);
- auto tracker_ptr = MemTracker::CreateTracker(-1, "BlockTest", nullptr, false);
- RowBatch row_batch(row_desc, 1024, tracker_ptr.get());
-
- int16_t k1 = -100;
- int32_t k2 = 100000;
- double k3 = 7.7;
-
- for (int i = 0; i < 1024; ++i, k1++, k2++, k3 += 0.1) {
- auto idx = row_batch.add_row();
- TupleRow* tuple_row = row_batch.get_row(idx);
-
- auto tuple = (Tuple*)(row_batch.tuple_data_pool()->allocate(tuple_desc->byte_size()));
- auto slot_desc = tuple_desc->slots()[0];
- memcpy(tuple->get_slot(slot_desc->tuple_offset()), &k1, column_descs[0].size);
- slot_desc = tuple_desc->slots()[1];
- memcpy(tuple->get_slot(slot_desc->tuple_offset()), &k2, column_descs[1].size);
- slot_desc = tuple_desc->slots()[2];
- memcpy(tuple->get_slot(slot_desc->tuple_offset()), &k3, column_descs[2].size);
-
- tuple_row->set_tuple(0, tuple);
- row_batch.commit_last_row();
- }
-
- auto block = row_batch.convert_to_vec_block();
- // 1. build arguments
- vectorized::ColumnNumbers arguments;
- arguments.emplace_back(block.get_position_by_name("k1"));
-
- doris::vectorized::ColumnsWithTypeAndName ctn = { block.get_by_position(block.get_position_by_name("k1")) };
- auto abs_function_ptr =
- doris::vectorized::SimpleFunctionFactory::instance().get_function("abs", ctn,
- std::make_shared<vectorized::DataTypeInt32>());
-
- // 2. build result column
- size_t num_columns_without_result = block.columns();
- block.insert({nullptr, block.get_by_position(0).type, "abs(k1)"});
-
- abs_function_ptr->execute(nullptr, block, arguments, num_columns_without_result, 1024, false);
-
- k1 = -100;
- for (int i = 0; i < 1024; ++i) {
- vectorized::ColumnPtr column = block.get_columns()[3];
- ASSERT_EQ(column->get_int(i), std::abs(k1++));
- }
-}
-
-} // namespace doris
-
-int main(int argc, char** argv) {
- ::testing::InitGoogleTest(&argc, argv);
- return RUN_ALL_TESTS();
-}
diff --git a/be/test/vec/function/function_arithmetic_test.cpp b/be/test/vec/function/function_arithmetic_test.cpp
index 3cf31b2..8dfe3d7 100644
--- a/be/test/vec/function/function_arithmetic_test.cpp
+++ b/be/test/vec/function/function_arithmetic_test.cpp
@@ -25,23 +25,17 @@
#include "util/url_coding.h"
#include "vec/core/field.h"
-namespace doris {
-
-using vectorized::Null;
-using vectorized::DataSet;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(function_arithmetic_test, function_arithmetic_mod_test) {
std::string func_name = "mod";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
DataSet data_set = {{{10, 1}, 0}, {{10, -2}, 0}, {{1234, 33}, 13}, {{1234, 0}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
@@ -49,19 +43,15 @@ TEST(function_arithmetic_test, function_arithmetic_divide_test) {
std::string func_name = "divide";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
DataSet data_set = {{{1234, 34}, 36.294117647058826}, {{1234, 0}, Null()}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64,
- vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Float64};
DataSet data_set = {{{1234.1, 34.6}, 35.667630057803464}, {{1234.34, 0.0}, Null()}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
}
@@ -69,7 +59,7 @@ TEST(function_arithmetic_test, bitnot_test) {
std::string func_name = "bitnot";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32};
DataSet data_set = {{{(int32_t)30}, ~(int32_t)30},
{{(int32_t)0}, ~(int32_t)0},
@@ -77,8 +67,7 @@ TEST(function_arithmetic_test, bitnot_test) {
{{(int32_t)-10.44}, ~(int32_t)-10},
{{(int32_t)-999.888}, ~(int32_t)-999}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
@@ -86,16 +75,14 @@ TEST(function_arithmetic_test, bitand_test) {
std::string func_name = "bitand";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
DataSet data_set = {{{(int32_t)30, (int32_t)12}, 30 & 12},
{{(int32_t)0, (int32_t)12}, 0 & 12},
{{(int32_t)-10, (int32_t)111}, -10 & 111},
{{(int32_t)-999, (int32_t)888}, -999 & 888}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
@@ -103,16 +90,14 @@ TEST(function_arithmetic_test, bitor_test) {
std::string func_name = "bitor";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
DataSet data_set = {{{(int32_t)30, (int32_t)12}, 30 | 12},
{{(int32_t)0, (int32_t)12}, 0 | 12},
{{(int32_t)-10, (int32_t)111}, -10 | 111},
{{(int32_t)-999, (int32_t)888}, -999 | 888}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
@@ -120,20 +105,18 @@ TEST(function_arithmetic_test, bitxor_test) {
std::string func_name = "bitxor";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
DataSet data_set = {{{(int32_t)30, (int32_t)12}, 30 ^ 12},
{{(int32_t)0, (int32_t)12}, 0 ^ 12},
{{(int32_t)-10, (int32_t)111}, -10 ^ 111},
{{(int32_t)-999, (int32_t)888}, -999 ^ 888}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
doris::CpuInfo::init();
diff --git a/be/test/vec/function/function_bitmap_test.cpp b/be/test/vec/function/function_bitmap_test.cpp
index 2b18a33..7a2c3c8 100644
--- a/be/test/vec/function/function_bitmap_test.cpp
+++ b/be/test/vec/function/function_bitmap_test.cpp
@@ -14,22 +14,17 @@
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.
-#include "vec/functions/function_totype.h"
-
#include <gtest/gtest.h>
-#include "util/bitmap_value.h"
#include "function_test_util.h"
+#include "util/bitmap_value.h"
+#include "vec/functions/function_totype.h"
-namespace doris {
-
-using vectorized::Null;
-using vectorized::DataSet;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(function_bitmap_test, function_bitmap_min_test) {
std::string func_name = "bitmap_min";
- std::vector<std::any> input_types = {TypeIndex::BitMap};
+ InputTypeSet input_types = {TypeIndex::BitMap};
auto bitmap1 = new BitmapValue(1);
auto bitmap2 = new BitmapValue(std::vector<uint64_t>({1, 9999999}));
@@ -39,12 +34,12 @@ TEST(function_bitmap_test, function_bitmap_min_test) {
{{empty_bitmap}, (int64_t)0},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
TEST(function_bitmap_test, function_bitmap_max_test) {
std::string func_name = "bitmap_max";
- std::vector<std::any> input_types = {TypeIndex::BitMap};
-
+ InputTypeSet input_types = {TypeIndex::BitMap};
+
auto bitmap1 = new BitmapValue(1);
auto bitmap2 = new BitmapValue(std::vector<uint64_t>({1, 9999999}));
auto empty_bitmap = new BitmapValue();
@@ -53,12 +48,12 @@ TEST(function_bitmap_test, function_bitmap_max_test) {
{{empty_bitmap}, (int64_t)0},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
TEST(function_bitmap_test, function_bitmap_to_string_test) {
std::string func_name = "bitmap_to_string";
- std::vector<std::any> input_types = {TypeIndex::BitMap};
+ InputTypeSet input_types = {TypeIndex::BitMap};
auto bitmap1 = new BitmapValue(1);
auto bitmap2 = new BitmapValue(std::vector<uint64_t>({1, 9999999}));
@@ -68,12 +63,12 @@ TEST(function_bitmap_test, function_bitmap_to_string_test) {
{{empty_bitmap}, std::string("")},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_bitmap_test, function_bitmap_and_count) {
std::string func_name = "bitmap_and_count";
- std::vector<std::any> input_types = {TypeIndex::BitMap, TypeIndex::BitMap};
+ InputTypeSet input_types = {TypeIndex::BitMap, TypeIndex::BitMap};
auto bitmap1 = new BitmapValue(std::vector<uint64_t>({1, 2, 3}));
auto bitmap2 = new BitmapValue(std::vector<uint64_t>({3, 4, 5}));
auto empty_bitmap = new BitmapValue();
@@ -81,12 +76,12 @@ TEST(function_bitmap_test, function_bitmap_and_count) {
{{bitmap1, bitmap1}, (int64_t)3},
{{bitmap1, bitmap2}, (int64_t)1}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
TEST(function_bitmap_test, function_bitmap_or_count) {
std::string func_name = "bitmap_or_count";
- std::vector<std::any> input_types = {TypeIndex::BitMap, TypeIndex::BitMap};
+ InputTypeSet input_types = {TypeIndex::BitMap, TypeIndex::BitMap};
auto bitmap1 = new BitmapValue(std::vector<uint64_t>({1, 2, 3}));
auto bitmap2 = new BitmapValue(std::vector<uint64_t>({1, 2, 3, 4}));
auto bitmap3 = new BitmapValue(std::vector<uint64_t>({2, 3}));
@@ -95,12 +90,12 @@ TEST(function_bitmap_test, function_bitmap_or_count) {
{{bitmap2, bitmap3}, (int64_t)4},
{{bitmap1, bitmap3}, (int64_t)3}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
TEST(function_bitmap_test, function_bitmap_xor_count) {
std::string func_name = "bitmap_xor_count";
- std::vector<std::any> input_types = {TypeIndex::BitMap, TypeIndex::BitMap};
+ InputTypeSet input_types = {TypeIndex::BitMap, TypeIndex::BitMap};
auto bitmap1 = new BitmapValue(std::vector<uint64_t>({1, 2, 3}));
auto bitmap2 = new BitmapValue(std::vector<uint64_t>({1, 2, 3, 4}));
auto bitmap3 = new BitmapValue(std::vector<uint64_t>({2, 3}));
@@ -110,9 +105,9 @@ TEST(function_bitmap_test, function_bitmap_xor_count) {
{{bitmap2, bitmap3}, (int64_t)2},
{{bitmap1, bitmap4}, (int64_t)2}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/be/test/vec/function/function_comparison_test.cpp b/be/test/vec/function/function_comparison_test.cpp
index 546ab12..8050f43 100644
--- a/be/test/vec/function/function_comparison_test.cpp
+++ b/be/test/vec/function/function_comparison_test.cpp
@@ -61,35 +61,36 @@ TEST(ComparisonTest, ComparisonFunctionTest) {
row_batch.commit_last_row();
}
- doris::vectorized::Block block = row_batch.convert_to_vec_block();
+ vectorized::Block block = row_batch.convert_to_vec_block();
// 1. compute the k1 > k2
- doris::vectorized::ColumnNumbers arguments;
+ vectorized::ColumnNumbers arguments;
arguments.emplace_back(block.get_position_by_name("k1"));
arguments.emplace_back(block.get_position_by_name("k2"));
size_t num_columns_without_result = block.columns();
- block.insert({nullptr, std::make_shared<doris::vectorized::DataTypeUInt8>(), "k1 > k2"});
+ block.insert({nullptr, std::make_shared<vectorized::DataTypeUInt8>(), "k1 > k2"});
- doris::vectorized::ColumnsWithTypeAndName ctn = { block.get_by_position(arguments[0]), block.get_by_position(arguments[1]) };
+ vectorized::ColumnsWithTypeAndName ctn = {block.get_by_position(arguments[0]),
+ block.get_by_position(arguments[1])};
- auto greater_function_ptr = doris::vectorized::SimpleFunctionFactory::instance().get_function("gt", ctn,
- std::make_shared<vectorized::DataTypeUInt8>());
- greater_function_ptr->execute(nullptr, block, arguments, num_columns_without_result, 1024, false);
-
+ auto greater_function_ptr = vectorized::SimpleFunctionFactory::instance().get_function(
+ "gt", ctn, std::make_shared<vectorized::DataTypeUInt8>());
+ greater_function_ptr->execute(nullptr, block, arguments, num_columns_without_result, 1024,
+ false);
k1 = -100;
k2 = 100;
for (int i = 0; i < 1024; ++i, k1++, k2--) {
- doris::vectorized::ColumnPtr column = block.get_columns()[3];
+ vectorized::ColumnPtr column = block.get_columns()[3];
ASSERT_EQ(column->get_bool(i), k1 > k2);
}
// 2. compute the k2 <= k3
num_columns_without_result = block.columns();
- block.insert({nullptr, std::make_shared<doris::vectorized::DataTypeUInt8>(), "k2 <= k3"});
+ block.insert({nullptr, std::make_shared<vectorized::DataTypeUInt8>(), "k2 <= k3"});
- auto less_or_equals_function_ptr =
- doris::vectorized::SimpleFunctionFactory::instance().get_function("le", ctn, std::make_shared<vectorized::DataTypeUInt8>());
+ auto less_or_equals_function_ptr = vectorized::SimpleFunctionFactory::instance().get_function(
+ "le", ctn, std::make_shared<vectorized::DataTypeUInt8>());
arguments[0] = 1;
arguments[1] = 2;
@@ -99,45 +100,44 @@ TEST(ComparisonTest, ComparisonFunctionTest) {
k2 = 100;
k3 = 7.7;
for (int i = 0; i < 1024; ++i, k3 += 0.1, k2--) {
- doris::vectorized::ColumnPtr column = block.get_columns()[4];
+ vectorized::ColumnPtr column = block.get_columns()[4];
ASSERT_EQ(column->get_bool(i), k2 <= k3);
}
num_columns_without_result = block.columns();
- block.insert({nullptr, std::make_shared<doris::vectorized::DataTypeUInt8>(),
- "k1 > k2 and k2 <= k3"});
+ block.insert({nullptr, std::make_shared<vectorized::DataTypeUInt8>(), "k1 > k2 and k2 <= k3"});
arguments[0] = 3;
arguments[1] = 4;
- doris::vectorized::ColumnsWithTypeAndName ctn2 = { block.get_by_position(arguments[0]), block.get_by_position(arguments[1]) };
- auto and_function_ptr = doris::vectorized::SimpleFunctionFactory::instance().get_function("and", ctn2,
- std::make_shared<vectorized::DataTypeUInt8>());
+ vectorized::ColumnsWithTypeAndName ctn2 = {block.get_by_position(arguments[0]),
+ block.get_by_position(arguments[1])};
+ auto and_function_ptr = vectorized::SimpleFunctionFactory::instance().get_function(
+ "and", ctn2, std::make_shared<vectorized::DataTypeUInt8>());
and_function_ptr->execute(nullptr, block, arguments, num_columns_without_result, 1024, false);
k1 = -100;
k2 = 100;
k3 = 7.7;
for (int i = 0; i < 1024; ++i, k1++, k3 += 0.1, k2--) {
- doris::vectorized::ColumnPtr column = block.get_columns()[5];
+ vectorized::ColumnPtr column = block.get_columns()[5];
ASSERT_EQ(column->get_bool(i), k1 > k2 and k2 <= k3);
}
num_columns_without_result = block.columns();
- block.insert(
- {nullptr, std::make_shared<doris::vectorized::DataTypeUInt8>(), "k1 > k2 or k2 <= k3"});
+ block.insert({nullptr, std::make_shared<vectorized::DataTypeUInt8>(), "k1 > k2 or k2 <= k3"});
arguments[0] = 3;
arguments[1] = 4;
- // doris::vectorized::ColumnsWithTypeAndName ctn2 = { block.get_by_position(arguments[0]), block.get_by_position(arguments[1]) };
- auto or_function_ptr = doris::vectorized::SimpleFunctionFactory::instance().get_function("or", ctn2,
- std::make_shared<vectorized::DataTypeUInt8>());
+ // vectorized::ColumnsWithTypeAndName ctn2 = { block.get_by_position(arguments[0]), block.get_by_position(arguments[1]) };
+ auto or_function_ptr = vectorized::SimpleFunctionFactory::instance().get_function(
+ "or", ctn2, std::make_shared<vectorized::DataTypeUInt8>());
or_function_ptr->execute(nullptr, block, arguments, num_columns_without_result, 1024, false);
k1 = -100;
k2 = 100;
k3 = 7.7;
for (int i = 0; i < 1024; ++i, k1++, k3 += 0.1, k2--) {
- doris::vectorized::ColumnPtr column = block.get_columns()[6];
+ vectorized::ColumnPtr column = block.get_columns()[6];
ASSERT_EQ(column->get_bool(i), k1 > k2 or k2 <= k3);
}
}
diff --git a/be/test/vec/function/function_hash_test.cpp b/be/test/vec/function/function_hash_test.cpp
index d5d41b2..41cfc52 100644
--- a/be/test/vec/function/function_hash_test.cpp
+++ b/be/test/vec/function/function_hash_test.cpp
@@ -22,45 +22,36 @@
#include "runtime/tuple_row.h"
#include "vec/functions/simple_function_factory.h"
-namespace doris {
-
-using vectorized::Null;
-using vectorized::DataSet;
+namespace doris::vectorized {
TEST(HashFunctionTest, murmur_hash_3_test) {
std::string func_name = "murmur_hash3_32";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{Null()}, Null()}, {{std::string("hello")}, (int32_t)1321743225}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String,
- vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("hello"), std::string("world")}, (int32_t)984713481},
{{std::string("hello"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String,
- vectorized::TypeIndex::String,
- vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("hello"), std::string("world"), std::string("!")},
(int32_t)-666935433},
{{std::string("hello"), std::string("world"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
};
}
@@ -68,42 +59,36 @@ TEST(HashFunctionTest, murmur_hash_2_test) {
std::string func_name = "murmurHash2_64";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{Null()}, Null()},
{{std::string("hello")}, (uint64_t)2191231550387646743ull}};
- vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeUInt64, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String,
- vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {
{{std::string("hello"), std::string("world")}, (uint64_t)11978658642541747642ull},
{{std::string("hello"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeUInt64, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String,
- vectorized::TypeIndex::String,
- vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("hello"), std::string("world"), std::string("!")},
(uint64_t)1367324781703025231ull},
{{std::string("hello"), std::string("world"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeUInt64, true>(func_name, input_types, data_set);
};
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/be/test/vec/function/function_ifnull_test.cpp b/be/test/vec/function/function_ifnull_test.cpp
index a775046..9d5df6c 100644
--- a/be/test/vec/function/function_ifnull_test.cpp
+++ b/be/test/vec/function/function_ifnull_test.cpp
@@ -25,59 +25,46 @@
#include "util/url_coding.h"
#include "vec/core/field.h"
-namespace doris {
-using vectorized::Null;
-using vectorized::DataSet;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(IfNullTest, Int_Test) {
std::string func_name = "ifnull";
- std::vector<std::any> input_types = {TypeIndex::Int32,
- TypeIndex::Int32};
- DataSet data_set = {
- {{4, 10}, 4},
- {{-4, 10}, -4},
- {{Null(), 5}, 5}};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
+ DataSet data_set = {{{4, 10}, 4}, {{-4, 10}, -4}, {{Null(), 5}, 5}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(IfNullTest, Float_Test) {
std::string func_name = "ifnull";
- std::vector<std::any> input_types = {TypeIndex::Float64,
- TypeIndex::Float64};
- DataSet data_set = {
- {{4.0, 10.0}, 4.0},
- {{-4.0, 10.0}, -4.0},
- {{Null(), 5.0}, 5.0}};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Float64};
+ DataSet data_set = {{{4.0, 10.0}, 4.0}, {{-4.0, 10.0}, -4.0}, {{Null(), 5.0}, 5.0}};
- vectorized::check_function<vectorized::DataTypeFloat64 , true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(IfNullTest, String_Test) {
std::string func_name = "ifnull";
- std::vector<std::any> input_types = {TypeIndex::String,
- TypeIndex::String};
- DataSet data_set = {
- {{std::string("4.0"), std::string("10.0")}, std::string("4.0")},
- {{std::string("hello"), std::string("10.0")}, std::string("hello")},
- {{Null(), std::string("world")}, std::string("world")}};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
+ DataSet data_set = {{{std::string("4.0"), std::string("10.0")}, std::string("4.0")},
+ {{std::string("hello"), std::string("10.0")}, std::string("hello")},
+ {{Null(), std::string("world")}, std::string("world")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(IfNullTest, String_Int_Test) {
std::string func_name = "ifnull";
- std::vector<std::any> input_types = {TypeIndex::DateTime,
- TypeIndex::DateTime};
- DataSet data_set = {
- {{std::string("2021-10-24 12:32:31"), std::string("2021-10-24 13:00:01")}, vectorized::str_to_data_time("2021-10-24 12:32:31")},
- {{Null(), std::string("2021-10-24 13:00:01")}, vectorized::str_to_data_time("2021-10-24 13:00:01")}};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::DateTime};
+ DataSet data_set = {{{std::string("2021-10-24 12:32:31"), std::string("2021-10-24 13:00:01")},
+ str_to_data_time("2021-10-24 12:32:31")},
+ {{Null(), std::string("2021-10-24 13:00:01")},
+ str_to_data_time("2021-10-24 13:00:01")}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types, data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/be/test/vec/function/function_like_test.cpp b/be/test/vec/function/function_like_test.cpp
index a2bf2b0..e27c479 100644
--- a/be/test/vec/function/function_like_test.cpp
+++ b/be/test/vec/function/function_like_test.cpp
@@ -23,100 +23,90 @@
#include "util/cpu_info.h"
#include "vec/core/types.h"
-namespace doris {
-
-using vectorized::DataSet;
-using vectorized::Null;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(FunctionLikeTest, like) {
std::string func_name = "like";
- DataSet data_set = {
- // sub_string
- {{std::string("abc"), std::string("%b%")}, uint8_t(1)},
- {{std::string("abc"), std::string("%ad%")}, uint8_t(0)},
- // end with
- {{std::string("abc"), std::string("%c")}, uint8_t(1)},
- {{std::string("ab"), std::string("%c")}, uint8_t(0)},
- // start with
- {{std::string("abc"), std::string("a%")}, uint8_t(1)},
- {{std::string("bc"), std::string("a%")}, uint8_t(0)},
- // equals
- {{std::string("abc"), std::string("abc")}, uint8_t(1)},
- {{std::string("abc"), std::string("ab")}, uint8_t(0)},
- // full regexp match
- {{std::string("abcd"), std::string("a_c%")}, uint8_t(1)},
- {{std::string("abcd"), std::string("a_d%")}, uint8_t(0)},
- {{std::string("abc"), std::string("__c")}, uint8_t(1)},
- {{std::string("abc"), std::string("_c")}, uint8_t(0)},
- {{std::string("abc"), std::string("_b_")}, uint8_t(1)},
- {{std::string("abc"), std::string("_a_")}, uint8_t(0)},
- {{std::string("abc"), std::string("a__")}, uint8_t(1)},
- {{std::string("abc"), std::string("a_")}, uint8_t(0)},
- // null
- {{std::string("abc"), Null()}, Null()},
- {{Null(), std::string("_x__ab%")}, Null()}
- };
+ DataSet data_set = {// sub_string
+ {{std::string("abc"), std::string("%b%")}, uint8_t(1)},
+ {{std::string("abc"), std::string("%ad%")}, uint8_t(0)},
+ // end with
+ {{std::string("abc"), std::string("%c")}, uint8_t(1)},
+ {{std::string("ab"), std::string("%c")}, uint8_t(0)},
+ // start with
+ {{std::string("abc"), std::string("a%")}, uint8_t(1)},
+ {{std::string("bc"), std::string("a%")}, uint8_t(0)},
+ // equals
+ {{std::string("abc"), std::string("abc")}, uint8_t(1)},
+ {{std::string("abc"), std::string("ab")}, uint8_t(0)},
+ // full regexp match
+ {{std::string("abcd"), std::string("a_c%")}, uint8_t(1)},
+ {{std::string("abcd"), std::string("a_d%")}, uint8_t(0)},
+ {{std::string("abc"), std::string("__c")}, uint8_t(1)},
+ {{std::string("abc"), std::string("_c")}, uint8_t(0)},
+ {{std::string("abc"), std::string("_b_")}, uint8_t(1)},
+ {{std::string("abc"), std::string("_a_")}, uint8_t(0)},
+ {{std::string("abc"), std::string("a__")}, uint8_t(1)},
+ {{std::string("abc"), std::string("a_")}, uint8_t(0)},
+ // null
+ {{std::string("abc"), Null()}, Null()},
+ {{Null(), std::string("_x__ab%")}, Null()}};
// pattern is constant value
- std::vector<std::any> const_pattern_input_types = {TypeIndex::String,
- vectorized::Consted {TypeIndex::String}};
- for (const auto& line: data_set) {
+ InputTypeSet const_pattern_input_types = {TypeIndex::String, Consted {TypeIndex::String}};
+ for (const auto& line : data_set) {
DataSet const_pattern_dataset = {line};
- vectorized::check_function<vectorized::DataTypeUInt8, true>(
- func_name, const_pattern_input_types, const_pattern_dataset);
+ check_function<DataTypeUInt8, true>(func_name, const_pattern_input_types,
+ const_pattern_dataset);
}
// pattern is not constant value
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
- vectorized::check_function<vectorized::DataTypeUInt8, true>(func_name, input_types, data_set);
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
+ check_function<DataTypeUInt8, true>(func_name, input_types, data_set);
}
TEST(FunctionLikeTest, regexp) {
std::string func_name = "regexp";
- DataSet data_set = {
- // sub_string
- {{std::string("abc"), std::string(".*b.*")}, uint8_t(1)},
- {{std::string("abc"), std::string(".*ad.*")}, uint8_t(0)},
- {{std::string("abc"), std::string(".*c")}, uint8_t(1)},
- {{std::string("abc"), std::string("a.*")}, uint8_t(1)},
- // end with
- {{std::string("abc"), std::string(".*c$")}, uint8_t(1)},
- {{std::string("ab"), std::string(".*c$")}, uint8_t(0)},
- // start with
- {{std::string("abc"), std::string("^a.*")}, uint8_t(1)},
- {{std::string("bc"), std::string("^a.*")}, uint8_t(0)},
- // equals
- {{std::string("abc"), std::string("^abc$")}, uint8_t(1)},
- {{std::string("abc"), std::string("^ab$")}, uint8_t(0)},
- // partial regexp match
- {{std::string("abcde"), std::string("a.*d")}, uint8_t(1)},
- {{std::string("abcd"), std::string("a.d")}, uint8_t(0)},
- {{std::string("abc"), std::string(".c")}, uint8_t(1)},
- {{std::string("abc"), std::string(".b.")}, uint8_t(1)},
- {{std::string("abc"), std::string(".a.")}, uint8_t(0)},
- // null
- {{std::string("abc"), Null()}, Null()},
- {{Null(), std::string("xxx.*")}, Null()}
- };
+ DataSet data_set = {// sub_string
+ {{std::string("abc"), std::string(".*b.*")}, uint8_t(1)},
+ {{std::string("abc"), std::string(".*ad.*")}, uint8_t(0)},
+ {{std::string("abc"), std::string(".*c")}, uint8_t(1)},
+ {{std::string("abc"), std::string("a.*")}, uint8_t(1)},
+ // end with
+ {{std::string("abc"), std::string(".*c$")}, uint8_t(1)},
+ {{std::string("ab"), std::string(".*c$")}, uint8_t(0)},
+ // start with
+ {{std::string("abc"), std::string("^a.*")}, uint8_t(1)},
+ {{std::string("bc"), std::string("^a.*")}, uint8_t(0)},
+ // equals
+ {{std::string("abc"), std::string("^abc$")}, uint8_t(1)},
+ {{std::string("abc"), std::string("^ab$")}, uint8_t(0)},
+ // partial regexp match
+ {{std::string("abcde"), std::string("a.*d")}, uint8_t(1)},
+ {{std::string("abcd"), std::string("a.d")}, uint8_t(0)},
+ {{std::string("abc"), std::string(".c")}, uint8_t(1)},
+ {{std::string("abc"), std::string(".b.")}, uint8_t(1)},
+ {{std::string("abc"), std::string(".a.")}, uint8_t(0)},
+ // null
+ {{std::string("abc"), Null()}, Null()},
+ {{Null(), std::string("xxx.*")}, Null()}};
// pattern is constant value
- std::vector<std::any> const_pattern_input_types = {TypeIndex::String,
- vectorized::Consted {TypeIndex::String}};
+ InputTypeSet const_pattern_input_types = {TypeIndex::String, Consted {TypeIndex::String}};
for (const auto& line : data_set) {
DataSet const_pattern_dataset = {line};
- vectorized::check_function<vectorized::DataTypeUInt8, true>(
- func_name, const_pattern_input_types, const_pattern_dataset);
+ check_function<DataTypeUInt8, true>(func_name, const_pattern_input_types,
+ const_pattern_dataset);
}
// pattern is not constant value
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
- vectorized::check_function<vectorized::DataTypeUInt8, true>(func_name, input_types, data_set);
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
+ check_function<DataTypeUInt8, true>(func_name, input_types, data_set);
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/be/test/vec/function/function_math_test.cpp b/be/test/vec/function/function_math_test.cpp
index 17748a0..eacfd01 100644
--- a/be/test/vec/function/function_math_test.cpp
+++ b/be/test/vec/function/function_math_test.cpp
@@ -27,15 +27,14 @@
#include "runtime/tuple_row.h"
#include "vec/functions/simple_function_factory.h"
-namespace doris {
+namespace doris::vectorized {
-using vectorized::Null;
-using vectorized::DataSet;
+using namespace ut_type;
TEST(MathFunctionTest, acos_test) {
std::string func_name = "acos"; //[-1,1] -->[0,pi]
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{-1.0}, 3.1415926535897931},
{{0.0}, M_PI / 2},
@@ -43,37 +42,37 @@ TEST(MathFunctionTest, acos_test) {
//{{3.14},nan("")},
{{1.0}, 0.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, asin_test) {
std::string func_name = "asin"; //[-1,1] -->[-pi_2, pi_2]
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {
{{-1.0}, -M_PI / 2}, {{0.0}, 0.0}, {{0.5}, 0.52359877559829893}, {{1.0}, M_PI / 2}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, atan_test) {
std::string func_name = "atan"; //[-,+] -->(pi_2,pi_2)
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{-1.0}, -0.78539816339744828},
{{0.0}, 0.0},
{{0.5}, 0.46364760900080609},
{{1.0}, 0.78539816339744828}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, cos_test) {
std::string func_name = "cos";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{-1.0}, 0.54030230586813977},
{{0.0}, 1.0},
@@ -81,13 +80,13 @@ TEST(MathFunctionTest, cos_test) {
{{M_PI}, -1.0},
{{1.0}, 0.54030230586813977}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, sin_test) {
std::string func_name = "sin";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{-1.0}, -0.8414709848078965},
{{0.0}, 0.0},
@@ -95,39 +94,39 @@ TEST(MathFunctionTest, sin_test) {
{{M_PI / 2}, 1.0},
{{1.0}, 0.8414709848078965}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, sqrt_test) {
std::string func_name = "sqrt"; //sqrt(x) x>=0
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{0.0}, 0.0},
{{2.0}, 1.4142135623730951},
{{9.0}, 3.0},
{{1000.0}, 31.622776601683793}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, tan_test) {
std::string func_name = "tan"; //tan(x)
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{0.0}, 0.0},
{{2.0}, -2.1850398632615189},
{{-1.0}, -1.5574077246549023},
{{1000.0}, 1.4703241557027185}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, exp_test) {
std::string func_name = "exp";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{-1.0}, 0.36787944117144233},
{{0.0}, 1.0},
@@ -135,158 +134,183 @@ TEST(MathFunctionTest, exp_test) {
{{-800.0}, 0.0},
{{1.0}, 2.7182818284590451}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, ln_test) {
std::string func_name = "ln"; // ln(x) x>0
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{1.0}, 0.0},
{{0.5}, -0.69314718055994529},
{{100.0}, 4.6051701859880918},
{{1000.0}, 6.9077552789821368}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, log2_test) {
std::string func_name = "log2"; // log2(x) x>0
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{1.0}, 0.0},
{{0.5}, -1.0},
{{100.0}, 6.6438561897747244},
{{1000.0}, 9.965784284662087}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, log10_test) {
std::string func_name = "log10"; // log10(x) x>0
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {
{{1.0}, 0.0}, {{0.5}, -0.3010299956639812}, {{100.0}, 2.0}, {{1000.0}, 3.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, log_test) {
std::string func_name = "log"; // log(x,y) x>0 y>0
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64,
- vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Float64};
DataSet data_set = {{{10.0, 1.0}, 0.0},
{{10.0, 100.0}, 2.0},
{{0.1, 5.0}, -0.69897000433601886},
{{2.0, 0.5}, -1.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, pow_test) {
std::string func_name = "pow"; // pow(x,y)
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64,
- vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Float64};
DataSet data_set = {{{10.0, 1.0}, 10.0},
{{10.0, 10.0}, 10000000000.0},
{{100.0, -2.0}, 0.0001},
{{2.0, 0.5}, 1.4142135623730951}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, truncate_test) {
std::string func_name = "truncate"; // truncate(x,y)
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64,
- vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Float64};
DataSet data_set = {{{123.4567, 3.0}, 123.456}, {{-123.4567, 3.0}, -123.456},
{{123.4567, 0.0}, 123.0}, {{-123.4567, 0.0}, -123.0},
{{123.4567, -2.0}, 100.0}, {{-123.4567, -2.0}, -100.0},
{{-123.4567, -3.0}, 0.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, ceil_test) {
std::string func_name = "ceil";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {
{{2.3}, (int64_t)3}, {{2.8}, (int64_t)3}, {{-2.3}, (int64_t)-2}, {{2.8}, (int64_t)3.0}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, floor_test) {
std::string func_name = "floor";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {
{{2.3}, (int64_t)2}, {{2.8}, (int64_t)2}, {{-2.3}, (int64_t)-3}, {{-2.8}, (int64_t)-3}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, degrees_test) {
std::string func_name = "degrees"; // degrees(x) rad-->C
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{1.0}, 57.295779513082323},
{{M_PI / 2}, 90.0},
{{0.0}, 0.0},
{{-2.0}, -114.59155902616465}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, radians_test) {
std::string func_name = "radians"; // radians(x) C--->rad
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{30.0}, 0.52359877559829882},
{{90.0}, M_PI / 2},
{{0.0}, 0.0},
{{-60.0}, -1.0471975511965976}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
+}
+
+TEST(MathFunctionTest, abs_test) {
+ std::string func_name = "abs";
+
+ {
+ InputTypeSet input_types = {TypeIndex::Float64};
+
+ DataSet data_set = {{{Null()}, Null()},
+ {{-0.0123}, 0.0123},
+ {{90.45}, 90.45},
+ {{0.0}, 0.0},
+ {{-60.0}, 60.0}};
+
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
+ }
+
+ {
+ InputTypeSet input_types = {TypeIndex::Int32};
+
+ DataSet data_set = {{{Null()}, Null()},
+ {{INT(3)}, BIGINT(3)},
+ {{INT(-3)}, BIGINT(3)},
+ {{INT(0)}, BIGINT(0)},
+ {{INT(-60)}, BIGINT(60)},
+ {{INT(INT_MAX)}, BIGINT(INT_MAX)},
+ {{INT(INT_MIN)}, BIGINT(-1ll * INT_MIN)}};
+
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
+ }
}
TEST(MathFunctionTest, positive_test) {
std::string func_name = "positive";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{0.0123}, 0.0123}, {{90.45}, 90.45}, {{0.0}, 0.0}, {{-60.0}, -60.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32};
DataSet data_set = {{{(int32_t)3}, (int32_t)3},
{{(int32_t)-3}, (int32_t)-3},
{{(int32_t)0}, (int32_t)0},
{{(int32_t)-60}, (int32_t)-60}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
@@ -294,24 +318,22 @@ TEST(MathFunctionTest, negative_test) {
std::string func_name = "negative";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{0.0123}, -0.0123}, {{90.45}, -90.45}, {{0.0}, 0.0}, {{-60.0}, 60.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32};
DataSet data_set = {{{(int32_t)3}, (int32_t)-3},
{{(int32_t)-3}, (int32_t)3},
{{(int32_t)0}, (int32_t)0},
{{(int32_t)-60}, (int32_t)60}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
}
@@ -319,22 +341,20 @@ TEST(MathFunctionTest, sign_test) {
std::string func_name = "sign"; // sign(x) // 1 0 -1
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32};
DataSet data_set = {{{(int32_t)30}, (int8_t)1.0},
{{(int32_t)0}, (int8_t)0.0},
{{(int32_t)-10}, (int8_t)-1.0}};
- vectorized::check_function<vectorized::DataTypeInt8, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt8, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{30.7}, (int8_t)1.0}, {{0.0}, (int8_t)0.0}, {{-10.6}, (int8_t)-1.0}};
- vectorized::check_function<vectorized::DataTypeInt8, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt8, true>(func_name, input_types, data_set);
}
}
@@ -342,43 +362,40 @@ TEST(MathFunctionTest, round_test) {
std::string func_name = "round"; // round(double) && round(double, int)
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
+ InputTypeSet input_types = {TypeIndex::Float64};
DataSet data_set = {{{30.1}, (int64_t)30}, {{90.6}, (int64_t)91}, {{Null()}, Null()},
{{0.0}, (int64_t)0}, {{-1.1}, (int64_t)-1}, {{-60.7}, (int64_t)-61}};
- vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt64, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Float64,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Int32};
DataSet data_set = {{{3.1415926, 2}, 3.14}, {{3.1415926, 3}, 3.142}, {{Null(), -2}, Null()},
{{193.0, -2}, 200.0}, {{193.0, -1}, 190.0}, {{193.0, -3}, 0.0}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
}
TEST(MathFunctionTest, bin_test) {
std::string func_name = "bin";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int64};
+ InputTypeSet input_types = {TypeIndex::Int64};
DataSet data_set = {{{(int64_t)10}, std::string("1010")},
{{(int64_t)1}, std::string("1")},
{{(int64_t)0}, std::string("0")},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, hex_test) {
std::string func_name = "hex"; // hex(int)
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int64};
+ InputTypeSet input_types = {TypeIndex::Int64};
DataSet data_set = {{{Null()}, Null()},
{{(int64_t)-1}, std::string("FFFFFFFFFFFFFFFF")},
@@ -390,12 +407,12 @@ TEST(MathFunctionTest, hex_test) {
{{(int64_t)9223372036854775807}, std::string("7FFFFFFFFFFFFFFF")},
{{(int64_t)-7453337203775808}, std::string("FFE5853AB393E6C0")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(MathFunctionTest, random_test) {
std::string func_name = "random"; // random(x)
- std::vector<std::any> input_types = {vectorized::Consted {vectorized::TypeIndex::Int64}};
+ InputTypeSet input_types = {Consted {TypeIndex::Int64}};
DataSet data_set = {{{Null()}, Null()},
{{(int64_t)0}, 0.15979336337046085},
{{(int64_t)10}, 0.60128310734097479},
@@ -405,12 +422,76 @@ TEST(MathFunctionTest, random_test) {
for (const auto& data : data_set) {
DataSet data_line = {data};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
- data_line);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_line);
+ }
+}
+
+TEST(MathFunctionTest, conv_test) {
+ std::string func_name = "conv";
+
+ {
+ InputTypeSet input_types = {TypeIndex::Int64, TypeIndex::Int8, TypeIndex::Int8};
+ DataSet data_set = {{{Null(), Null(), Null()}, Null()},
+ {{BIGINT(230), TINYINT(10), TINYINT(16)}, VARCHAR("E6")},
+ {{BIGINT(15), TINYINT(10), TINYINT(2)}, VARCHAR("1111")}};
+
+ for (const auto& data : data_set) {
+ DataSet data_line = {data};
+ check_function<DataTypeString, true>(func_name, input_types, data_line);
+ }
+ }
+
+ {
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int8, TypeIndex::Int8};
+ DataSet data_set = {{{Null(), Null(), Null()}, Null()},
+ {{VARCHAR("ff"), TINYINT(16), TINYINT(10)}, VARCHAR("255")}};
+
+ for (const auto& data : data_set) {
+ DataSet data_line = {data};
+ check_function<DataTypeString, true>(func_name, input_types, data_line);
+ }
+ }
+}
+
+TEST(MathFunctionTest, money_format_test) {
+ std::string func_name = "money_format";
+
+ {
+ InputTypeSet input_types = {TypeIndex::Int64};
+ DataSet data_set = {{{Null()}, Null()},
+ {{BIGINT(17014116)}, VARCHAR("17,014,116.00")},
+ {{BIGINT(-17014116)}, VARCHAR("-17,014,116.00")}};
+
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
+ }
+
+ {
+ InputTypeSet input_types = {TypeIndex::Int128};
+ DataSet data_set = {{{Null()}, Null()},
+ {{LARGEINT(17014116)}, VARCHAR("17,014,116.00")},
+ {{LARGEINT(-17014116)}, VARCHAR("-17,014,116.00")}};
+
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
+ }
+ {
+ InputTypeSet input_types = {TypeIndex::Float64};
+ DataSet data_set = {{{Null()}, Null()},
+ {{DOUBLE(17014116.67)}, VARCHAR("17,014,116.67")},
+ {{DOUBLE(-17014116.67)}, VARCHAR("-17,014,116.67")}};
+
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
+ }
+ {
+ InputTypeSet input_types = {TypeIndex::Decimal128};
+ DataSet data_set = {{{Null()}, Null()},
+ {{DECIMAL(17014116.67)}, VARCHAR("17,014,116.67")},
+ {{DECIMAL(-17014116.67)}, VARCHAR("-17,014,116.67")}};
+
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/be/test/vec/function/function_nullif_test.cpp b/be/test/vec/function/function_nullif_test.cpp
index b53e7f2..73d0ca9 100644
--- a/be/test/vec/function/function_nullif_test.cpp
+++ b/be/test/vec/function/function_nullif_test.cpp
@@ -25,49 +25,38 @@
#include "util/url_coding.h"
#include "vec/core/field.h"
-namespace doris {
-using vectorized::Null;
-using vectorized::DataSet;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(NullIfTest, Int_Test) {
std::string func_name = "nullif";
- std::vector<std::any> input_types = {TypeIndex::Int32,
- TypeIndex::Int32};
- DataSet data_set = {
- {{4, 10}, 4},
- {{-4, -4}, Null()},
- {{5, Null()}, 5}};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
+ DataSet data_set = {{{4, 10}, 4}, {{-4, -4}, Null()}, {{5, Null()}, 5}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(NullIfTest, Float_Test) {
std::string func_name = "nullif";
- std::vector<std::any> input_types = {TypeIndex::Float64,
- TypeIndex::Float64};
- DataSet data_set = {
- {{4.0, 10.0}, 4.0},
- {{-4.0, -4.0}, Null()},
- {{5.0, Null()}, 5.0}};
+ InputTypeSet input_types = {TypeIndex::Float64, TypeIndex::Float64};
+ DataSet data_set = {{{4.0, 10.0}, 4.0}, {{-4.0, -4.0}, Null()}, {{5.0, Null()}, 5.0}};
- vectorized::check_function<vectorized::DataTypeFloat64 , true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
-
TEST(NullIfTest, String_Int_Test) {
std::string func_name = "nullif";
- std::vector<std::any> input_types = {TypeIndex::DateTime,
- TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::DateTime};
DataSet data_set = {
- {{std::string("2021-10-24 12:32:31"), std::string("2021-10-24 13:00:01")}, vectorized::str_to_data_time("2021-10-24 12:32:31")},
+ {{std::string("2021-10-24 12:32:31"), std::string("2021-10-24 13:00:01")},
+ str_to_data_time("2021-10-24 12:32:31")},
{{std::string("2021-10-24 13:00:01"), std::string("2021-10-24 13:00:01")}, Null()},
- {{std::string("2021-10-24 13:00:01"), Null()}, vectorized::str_to_data_time("2021-10-24 13:00:01")}};
+ {{std::string("2021-10-24 13:00:01"), Null()},
+ str_to_data_time("2021-10-24 13:00:01")}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types, data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/be/test/vec/function/function_string_test.cpp b/be/test/vec/function/function_string_test.cpp
index 9f1f5d7..5617a36 100644
--- a/be/test/vec/function/function_string_test.cpp
+++ b/be/test/vec/function/function_string_test.cpp
@@ -26,30 +26,43 @@
#include "util/url_coding.h"
#include "vec/core/field.h"
-namespace doris {
-
-using vectorized::Null;
-using vectorized::DataSet;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(function_string_test, function_string_substr_test) {
std::string func_name = "substr";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::Int32, TypeIndex::Int32};
- DataSet data_set = {
- {{std::string("asd你好"), 4, 10}, std::string("\xE4\xBD\xA0\xE5\xA5\xBD")}, //你好
- {{std::string("hello word"), -5, 5}, std::string(" word")},
- {{std::string("hello word"), 1, 12}, std::string("hello word")},
- {{std::string("HELLO,!^%"), 4, 2}, std::string("LO")},
- {{std::string(""), 5, 4}, Null()},
- {{Null(), 5, 4}, Null()}};
+ {
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32, TypeIndex::Int32};
+
+ DataSet data_set = {
+ {{std::string("asd你好"), 4, 10}, std::string("\xE4\xBD\xA0\xE5\xA5\xBD")}, //你好
+ {{std::string("hello word"), -5, 5}, std::string(" word")},
+ {{std::string("hello word"), 1, 12}, std::string("hello word")},
+ {{std::string("HELLO,!^%"), 4, 2}, std::string("LO")},
+ {{std::string(""), 5, 4}, Null()},
+ {{Null(), 5, 4}, Null()}};
+
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
+ }
+
+ {
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ DataSet data_set = {
+ {{std::string("asd你好"), 4}, std::string("\xE4\xBD\xA0\xE5\xA5\xBD")}, //你好
+ {{std::string("hello word"), -5}, std::string(" word")},
+ {{std::string("hello word"), 1}, std::string("hello word")},
+ {{std::string("HELLO,!^%"), 4}, std::string("LO,!^%")},
+ {{std::string(""), 5, 4}, Null()},
+ {{Null(), 5, 4}, Null()}};
+
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
+ }
}
TEST(function_string_test, function_string_strright_test) {
std::string func_name = "strright";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32};
DataSet data_set = {{{std::string("asd"), 1}, std::string("d")},
{{std::string("hello word"), -2}, std::string("ello word")},
@@ -58,12 +71,12 @@ TEST(function_string_test, function_string_strright_test) {
{{std::string(""), 3}, std::string("")},
{{Null(), 3}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_strleft_test) {
std::string func_name = "strleft";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32};
DataSet data_set = {{{std::string("asd"), 1}, std::string("a")},
{{std::string("hel lo "), 5}, std::string("hel ")},
@@ -72,36 +85,36 @@ TEST(function_string_test, function_string_strleft_test) {
{{std::string(""), 2}, Null()},
{{Null(), 3}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_lower_test) {
std::string func_name = "lower";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("ASD")}, std::string("asd")},
{{std::string("HELLO123")}, std::string("hello123")},
{{std::string("MYtestSTR")}, std::string("myteststr")},
{{std::string("HELLO,!^%")}, std::string("hello,!^%")},
{{std::string("")}, std::string("")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_upper_test) {
std::string func_name = "upper";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("asd")}, std::string("ASD")},
{{std::string("hello123")}, std::string("HELLO123")},
{{std::string("HELLO,!^%")}, std::string("HELLO,!^%")},
{{std::string("MYtestStr")}, std::string("MYTESTSTR")},
{{std::string("")}, std::string("")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_trim_test) {
std::string func_name = "trim";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("a sd")}, std::string("a sd")},
{{std::string(" hello 123 ")}, std::string("hello 123")},
{{std::string(" HELLO,!^%")}, std::string("HELLO,!^%")},
@@ -109,35 +122,35 @@ TEST(function_string_test, function_string_trim_test) {
{{Null()}, Null()},
{{std::string("")}, std::string("")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_ltrim_test) {
std::string func_name = "ltrim";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {
{{std::string("a sd")}, std::string("a sd")},
{{std::string(" hello 123 ")}, std::string("hello 123 ")},
{{std::string(" HELLO,!^%")}, std::string("HELLO,!^%")},
{{std::string(" 你好MY test Str你好 ")}, std::string("你好MY test Str你好 ")},
{{std::string("")}, std::string("")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_rtrim_test) {
std::string func_name = "rtrim";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("a sd ")}, std::string("a sd")},
{{std::string("hello 123 ")}, std::string("hello 123")},
{{std::string(" HELLO,!^%")}, std::string(" HELLO,!^%")},
{{std::string(" MY test Str你好 ")}, std::string(" MY test Str你好")},
{{std::string("")}, std::string("")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_repeat_test) {
std::string func_name = "repeat";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32};
DataSet data_set = {{{std::string("a"), 3}, std::string("aaa")},
{{std::string("hel lo"), 2}, std::string("hel lohel lo")},
@@ -145,50 +158,50 @@ TEST(function_string_test, function_string_repeat_test) {
{{std::string(""), 1}, std::string("")},
{{std::string("HELLO,!^%"), 2}, std::string("HELLO,!^%HELLO,!^%")},
{{std::string("你"), 2}, std::string("你你")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_reverse_test) {
std::string func_name = "reverse";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("asd ")}, std::string(" dsa")},
{{std::string(" hello 123 ")}, std::string(" 321 olleh ")},
{{std::string(" HELLO,!^%")}, std::string("%^!,OLLEH ")},
{{std::string("你好啊")}, std::string("啊好你")},
{{std::string("")}, std::string("")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_length_test) {
std::string func_name = "length";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("asd ")}, int32_t(4)},
{{std::string(" hello 123 ")}, int32_t(13)},
{{std::string(" HELLO,!^%")}, int32_t(11)},
{{std::string("你好啊")}, int32_t(9)},
{{std::string("")}, int32_t(0)}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_append_trailing_char_if_absent_test) {
std::string func_name = "append_trailing_char_if_absent";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("ASD"), std::string("D")}, std::string("ASD")},
{{std::string("AS"), std::string("D")}, std::string("ASD")},
{{std::string(""), std::string("")}, Null()},
{{std::string(""), std::string("A")}, std::string("A")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_starts_with_test) {
std::string func_name = "starts_with";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("hello world"), std::string("hello")}, uint8_t(1)},
{{std::string("hello world"), std::string("world")}, uint8_t(0)},
@@ -197,13 +210,13 @@ TEST(function_string_test, function_starts_with_test) {
{{std::string("你好"), Null()}, Null()},
{{Null(), std::string("")}, Null()}};
- vectorized::check_function<vectorized::DataTypeUInt8, true>(func_name, input_types, data_set);
+ check_function<DataTypeUInt8, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_ends_with_test) {
std::string func_name = "ends_with";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("hello world"), std::string("hello")}, uint8_t(0)},
{{std::string("hello world"), std::string("world")}, uint8_t(1)},
@@ -212,13 +225,13 @@ TEST(function_string_test, function_ends_with_test) {
{{std::string("你好"), Null()}, Null()},
{{Null(), std::string("")}, Null()}};
- vectorized::check_function<vectorized::DataTypeUInt8, true>(func_name, input_types, data_set);
+ check_function<DataTypeUInt8, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_lpad_test) {
std::string func_name = "lpad";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::Int32, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32, TypeIndex::String};
DataSet data_set = {{{std::string("hi"), 5, std::string("?")}, std::string("???hi")},
{{std::string("g8%7IgY%AHx7luNtf8Kh"), 20, std::string("")},
@@ -233,13 +246,13 @@ TEST(function_string_test, function_lpad_test) {
{{std::string("hi"), 5, std::string("呵呵")}, std::string("呵呵呵hi")},
{{std::string("呵呵"), 5, std::string("hi")}, std::string("hih呵呵")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_rpad_test) {
std::string func_name = "rpad";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::Int32, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::Int32, TypeIndex::String};
DataSet data_set = {{{std::string("hi"), 5, std::string("?")}, std::string("hi???")},
{{std::string("g8%7IgY%AHx7luNtf8Kh"), 20, std::string("")},
@@ -254,62 +267,59 @@ TEST(function_string_test, function_rpad_test) {
{{std::string("hi"), 5, std::string("呵呵")}, std::string("hi呵呵呵")},
{{std::string("呵呵"), 5, std::string("hi")}, std::string("呵呵hih")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_ascii_test) {
std::string func_name = "ascii";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("")}, 0},
{{std::string("aa")}, 97},
{{std::string("我")}, 230},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_char_length_test) {
std::string func_name = "char_length";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("")}, 0}, {{std::string("aa")}, 2},
{{std::string("我")}, 1}, {{std::string("我a")}, 2},
{{std::string("a我")}, 2}, {{std::string("123")}, 3},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_concat_test) {
std::string func_name = "concat";
{
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("")}, std::string("")},
{{std::string("123")}, std::string("123")},
{{Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string(""), std::string("")}, std::string("")},
{{std::string("123"), std::string("45")}, std::string("12345")},
{{std::string("123"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String,
- TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {
{{std::string(""), std::string("1"), std::string("")}, std::string("1")},
@@ -317,15 +327,14 @@ TEST(function_string_test, function_concat_test) {
std::string("123456789")},
{{std::string("123"), Null(), std::string("789")}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
};
}
TEST(function_string_test, function_concat_ws_test) {
std::string func_name = "concat_ws";
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("-"), std::string("")}, std::string("")},
{{std::string(""), std::string("123")}, std::string("123")},
@@ -333,13 +342,11 @@ TEST(function_string_test, function_concat_ws_test) {
{{Null(), std::string("")}, Null()},
{{Null(), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String,
- TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {
{{std::string("-"), std::string(""), std::string("")}, std::string("-")},
@@ -348,13 +355,12 @@ TEST(function_string_test, function_concat_ws_test) {
{{Null(), std::string(""), std::string("")}, Null()},
{{Null(), std::string(""), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
};
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String,
- TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String,
+ TypeIndex::String};
DataSet data_set = {
{{std::string("-"), std::string(""), std::string(""), std::string("")},
@@ -367,27 +373,26 @@ TEST(function_string_test, function_concat_ws_test) {
{{std::string("-"), std::string("123"), Null(), std::string("456")},
std::string("123-456")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
};
}
TEST(function_string_test, function_null_or_empty_test) {
std::string func_name = "null_or_empty";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("")}, uint8(true)},
{{std::string("aa")}, uint8(false)},
{{std::string("我")}, uint8(false)},
{{Null()}, uint8(true)}};
- vectorized::check_function<vectorized::DataTypeUInt8, false>(func_name, input_types, data_set);
+ check_function<DataTypeUInt8, false>(func_name, input_types, data_set);
}
TEST(function_string_test, function_to_base64_test) {
std::string func_name = "to_base64";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("asd你好")}, {std::string("YXNk5L2g5aW9")}},
{{std::string("hello world")}, {std::string("aGVsbG8gd29ybGQ=")}},
@@ -396,12 +401,12 @@ TEST(function_string_test, function_to_base64_test) {
{{std::string("MYtestSTR")}, {std::string("TVl0ZXN0U1RS")}},
{{std::string("ò&ø")}, {std::string("w7Imw7g=")}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_from_base64_test) {
std::string func_name = "from_base64";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{std::string("YXNk5L2g5aW9")}, {std::string("asd你好")}},
{{std::string("aGVsbG8gd29ybGQ=")}, {std::string("hello world")}},
@@ -412,12 +417,12 @@ TEST(function_string_test, function_from_base64_test) {
{{std::string("ò&ø")}, {Null()}},
{{std::string("你好哈喽")}, {Null()}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_reverse_test) {
std::string func_name = "reverse";
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {
{{std::string("")}, {std::string("")}},
{{std::string("a")}, {std::string("a")}},
@@ -427,13 +432,13 @@ TEST(function_string_test, function_reverse_test) {
{{std::string("A攀c")}, {std::string("c攀A")}},
{{std::string("NULL")}, {std::string("LLUN")}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_instr_test) {
std::string func_name = "instr";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("abcdefg"), std::string("efg")}, 5},
{{std::string("aa"), std::string("a")}, 1},
@@ -443,13 +448,13 @@ TEST(function_string_test, function_instr_test) {
{{std::string(""), std::string("")}, 1},
{{std::string("aaaab"), std::string("bb")}, 0}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_find_in_set_test) {
std::string func_name = "find_in_set";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("abcdefg"), std::string("a,b,c")}, 0},
{{std::string("aa"), std::string("a,aa,aaa")}, 2},
@@ -459,12 +464,12 @@ TEST(function_string_test, function_find_in_set_test) {
{{std::string("a"), std::string("")}, 0},
{{std::string(""), std::string(",,")}, 1}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_string_splitpart_test) {
std::string func_name = "split_part";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::Int32};
DataSet data_set = {
{{std::string("prefix_string1"), std::string("_"), 2}, std::string("string1")},
@@ -476,14 +481,14 @@ TEST(function_string_test, function_string_splitpart_test) {
{{std::string("prefix_string"), std::string("__"), Null()}, Null()},
{{std::string("prefix_string"), std::string("__"), -1}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_md5sum_test) {
std::string func_name = "md5sum";
{
- std::vector<std::any> input_types = {TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {
{{std::string("asd你好")}, {std::string("a38c15675555017e6b8ea042f2eb24f5")}},
{{std::string("hello world")}, {std::string("5eb63bbbe01eeed093cb22bb8f5acdc3")}},
@@ -494,12 +499,11 @@ TEST(function_string_test, function_md5sum_test) {
{{std::string("MYtestSTR")}, {std::string("cd24c90b3fc1192eb1879093029e87d4")}},
{{std::string("ò&ø")}, {std::string("fd157b4cb921fa91acc667380184d59c")}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("asd"), std::string("你好")},
{std::string("a38c15675555017e6b8ea042f2eb24f5")}},
{{std::string("hello "), std::string("world")},
@@ -508,13 +512,11 @@ TEST(function_string_test, function_md5sum_test) {
{std::string("b8e6e34d1cc3dc76b784ddfdfb7df800")}},
{{Null(), std::string("HELLO")}, {Null()}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String,
- TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {{{std::string("a"), std::string("sd"), std::string("你好")},
{std::string("a38c15675555017e6b8ea042f2eb24f5")}},
{{std::string(""), std::string(""), std::string("")},
@@ -523,15 +525,14 @@ TEST(function_string_test, function_md5sum_test) {
{std::string("b8e6e34d1cc3dc76b784ddfdfb7df800")}},
{{Null(), std::string("HELLO"), Null()}, {Null()}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
}
TEST(function_string_test, function_aes_encrypt_test) {
std::string func_name = "aes_encrypt";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
const char* key = "doris";
const char* src[6] = {"aaaaaa", "bbbbbb", "cccccc", "dddddd", "eeeeee", ""};
@@ -541,9 +542,9 @@ TEST(function_string_test, function_aes_encrypt_test) {
int cipher_len = strlen(src[i]) + 16;
char p[cipher_len];
- int outlen =
- EncryptionUtil::encrypt(AES_128_ECB, (unsigned char*)src[i], strlen(src[i]),
- (unsigned char*)key, strlen(key), NULL, true, (unsigned char*)p);
+ int outlen = EncryptionUtil::encrypt(AES_128_ECB, (unsigned char*)src[i], strlen(src[i]),
+ (unsigned char*)key, strlen(key), NULL, true,
+ (unsigned char*)p);
r[i] = std::string(p, outlen);
}
@@ -555,13 +556,13 @@ TEST(function_string_test, function_aes_encrypt_test) {
{{std::string(src[5]), std::string(key)}, Null()},
{{Null(), std::string(key)}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_aes_decrypt_test) {
std::string func_name = "aes_decrypt";
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
const char* key = "doris";
const char* src[5] = {"aaaaaa", "bbbbbb", "cccccc", "dddddd", "eeeeee"};
@@ -571,9 +572,9 @@ TEST(function_string_test, function_aes_decrypt_test) {
int cipher_len = strlen(src[i]) + 16;
char p[cipher_len];
- int outlen =
- EncryptionUtil::encrypt(AES_128_ECB, (unsigned char*)src[i], strlen(src[i]),
- (unsigned char*)key, strlen(key), NULL, true, (unsigned char*)p);
+ int outlen = EncryptionUtil::encrypt(AES_128_ECB, (unsigned char*)src[i], strlen(src[i]),
+ (unsigned char*)key, strlen(key), NULL, true,
+ (unsigned char*)p);
r[i] = std::string(p, outlen);
}
@@ -584,14 +585,14 @@ TEST(function_string_test, function_aes_decrypt_test) {
{{r[4], std::string(key)}, std::string(src[4])},
{{Null(), std::string(key)}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_parse_url_test) {
std::string func_name = "parse_url";
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String};
DataSet data_set = {
{{std::string("zhangsan"), std::string("HOST")}, {Null()}},
{{std::string("facebook.com/path/p1"), std::string("HOST")}, {Null()}},
@@ -625,13 +626,11 @@ TEST(function_string_test, function_parse_url_test) {
{{std::string("http://fb.com/path/p1.p?q=1#f"), std::string("QUERY")},
{std::string("q=1")}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {TypeIndex::String, TypeIndex::String,
- TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {
{{std::string("http://fb.com/path/p1.p?q=1#f"), std::string("QUERY"),
std::string("q")},
@@ -645,14 +644,13 @@ TEST(function_string_test, function_parse_url_test) {
std::string("q")},
{Null()}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
}
TEST(function_string_test, function_hex_test) {
std::string func_name = "hex";
- std::vector<std::any> input_types = {vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{Null()}, Null()},
{{std::string("0")}, std::string("30")},
{{std::string("1")}, std::string("31")},
@@ -663,12 +661,12 @@ TEST(function_string_test, function_hex_test) {
{{std::string("我")}, std::string("E68891")},
{{std::string("?")}, std::string("3F")},
{{std::string("？")}, std::string("EFBC9F")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_unhex_test) {
std::string func_name = "unhex";
- std::vector<std::any> input_types = {vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String};
DataSet data_set = {{{Null()}, {Null()}},
{{std::string("@!#")}, std::string("")},
{{std::string("")}, std::string("")},
@@ -678,52 +676,43 @@ TEST(function_string_test, function_unhex_test) {
{{std::string("41")}, std::string("A")},
{{std::string("313233")}, std::string("123")},
{{std::string("EFBC9F")}, std::string("？")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(function_string_test, function_coalesce_test) {
std::string func_name = "coalesce";
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32, TypeIndex::Int32};
DataSet data_set = {{{Null(), Null(), (int32_t)1}, {(int32_t)1}},
{{Null(), Null(), (int32_t)2}, {(int32_t)2}},
{{Null(), Null(), (int32_t)3}, {(int32_t)3}},
{{Null(), Null(), (int32_t)4}, {(int32_t)4}}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String,
- vectorized::TypeIndex::String,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::Int32};
DataSet data_set = {
{{std::string("qwer"), Null(), (int32_t)1}, {std::string("qwer")}},
{{std::string("asdf"), Null(), (int32_t)2}, {std::string("asdf")}},
{{std::string("zxcv"), Null(), (int32_t)3}, {std::string("zxcv")}},
{{std::string("vbnm"), Null(), (int32_t)4}, {std::string("vbnm")}},
};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
- std::vector<std::any> input_types = {vectorized::TypeIndex::String,
- vectorized::TypeIndex::String,
- vectorized::TypeIndex::String};
+ InputTypeSet input_types = {TypeIndex::String, TypeIndex::String, TypeIndex::String};
DataSet data_set = {
{{Null(), std::string("abc"), std::string("hij")}, {std::string("abc")}},
{{Null(), std::string("def"), std::string("klm")}, {std::string("def")}},
{{Null(), std::string(""), std::string("xyz")}, {std::string("")}},
{{Null(), Null(), std::string("uvw")}, {std::string("uvw")}}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
}
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
doris::CpuInfo::init();
diff --git a/be/test/vec/function/function_test_util.h b/be/test/vec/function/function_test_util.h
index 53ee319..d543178 100644
--- a/be/test/vec/function/function_test_util.h
+++ b/be/test/vec/function/function_test_util.h
@@ -26,18 +26,36 @@
#include "runtime/row_batch.h"
#include "runtime/tuple_row.h"
#include "testutil/function_utils.h"
-#include "util/bitmap_value.h"
#include "udf/udf.h"
#include "udf/udf_internal.h"
+#include "util/bitmap_value.h"
#include "vec/columns/column_complex.h"
#include "vec/functions/function_string.h"
#include "vec/functions/function_string_to_string.h"
#include "vec/functions/simple_function_factory.h"
#include "vec/runtime/vdatetime_value.h"
-namespace doris {
-namespace vectorized {
+
+namespace doris::vectorized {
using DataSet = std::vector<std::pair<std::vector<std::any>, std::any>>;
+using InputTypeSet = std::vector<std::any>;
+
+namespace ut_type {
+using TINYINT = int8_t;
+using SMALLINT = int16_t;
+using INT = int32_t;
+using BIGINT = int64_t;
+using LARGEINT = int128_t;
+
+using VARCHAR = std::string;
+using CHAR = std::string;
+using STRING = std::string;
+
+using DOUBLE = double;
+using FLOAT = float;
+inline auto DECIMAL = Decimal<Int128>::double_to_decimal;
+} // namespace ut_type
+
int64_t str_to_data_time(std::string datetime_str, bool data_time = true) {
VecDateTimeValue v;
v.from_date_str(datetime_str.c_str(), datetime_str.size());
@@ -46,7 +64,7 @@ int64_t str_to_data_time(std::string datetime_str, bool data_time = true) {
} else {
v.cast_to_date();
}
- return binary_cast<doris::vectorized::VecDateTimeValue, Int64>(v);
+ return binary_cast<VecDateTimeValue, Int64>(v);
}
template <typename ColumnType, typename Column, typename NullColumn>
@@ -62,8 +80,8 @@ void insert_column_to_block(std::list<ColumnPtr>& columns, ColumnsWithTypeAndNam
ctn.emplace_back(type_and_name);
}
-// Null values are represented by vectorized::Null()
-// The type of the constant column is represented as follows: vectorized::Consted {vectorized::TypeIndex::String}
+// Null values are represented by Null()
+// The type of the constant column is represented as follows: Consted {TypeIndex::String}
// A DataSet with a constant column can only have one row of data
template <typename ReturnType, bool nullable = false>
@@ -105,7 +123,7 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
col->insert_default();
continue;
}
- auto str = std::any_cast<std::string>(data_set[j].first[i]);
+ auto str = std::any_cast<ut_type::STRING>(data_set[j].first[i]);
col->insert_data(str.c_str(), str.size());
}
insert_column_to_block<DataTypeString>(columns, ctn, std::move(col),
@@ -127,6 +145,36 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
std::move(null_map), block, col_name, i,
is_const, row_size);
arg_type.type = doris_udf::FunctionContext::TYPE_OBJECT;
+ } else if (tp == TypeIndex::Int8) {
+ auto col = ColumnInt8::create();
+
+ for (int j = 0; j < row_size; j++) {
+ if (data_set[j].first[i].type() == typeid(Null)) {
+ null_map_data[j] = true;
+ col->insert_default();
+ continue;
+ }
+ auto value = std::any_cast<ut_type::TINYINT>(data_set[j].first[i]);
+ col->insert_data(reinterpret_cast<char*>(&value), 0);
+ }
+ insert_column_to_block<DataTypeInt8>(columns, ctn, std::move(col), std::move(null_map),
+ block, col_name, i, is_const, row_size);
+ arg_type.type = doris_udf::FunctionContext::TYPE_TINYINT;
+ } else if (tp == TypeIndex::Int16) {
+ auto col = ColumnInt16::create();
+
+ for (int j = 0; j < row_size; j++) {
+ if (data_set[j].first[i].type() == typeid(Null)) {
+ null_map_data[j] = true;
+ col->insert_default();
+ continue;
+ }
+ auto value = std::any_cast<ut_type::SMALLINT>(data_set[j].first[i]);
+ col->insert_data(reinterpret_cast<char*>(&value), 0);
+ }
+ insert_column_to_block<DataTypeInt16>(columns, ctn, std::move(col), std::move(null_map),
+ block, col_name, i, is_const, row_size);
+ arg_type.type = doris_udf::FunctionContext::TYPE_SMALLINT;
} else if (tp == TypeIndex::Int32) {
auto col = ColumnInt32::create();
@@ -136,7 +184,7 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
col->insert_default();
continue;
}
- auto value = std::any_cast<int>(data_set[j].first[i]);
+ auto value = std::any_cast<ut_type::INT>(data_set[j].first[i]);
col->insert_data(reinterpret_cast<char*>(&value), 0);
}
insert_column_to_block<DataTypeInt32>(columns, ctn, std::move(col), std::move(null_map),
@@ -151,28 +199,60 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
col->insert_default();
continue;
}
- auto value = std::any_cast<int64>(data_set[j].first[i]);
+ auto value = std::any_cast<ut_type::BIGINT>(data_set[j].first[i]);
col->insert_data(reinterpret_cast<char*>(&value), 0);
}
insert_column_to_block<DataTypeInt64>(columns, ctn, std::move(col), std::move(null_map),
block, col_name, i, is_const, row_size);
arg_type.type = doris_udf::FunctionContext::TYPE_BIGINT;
+ } else if (tp == TypeIndex::Int128) {
+ auto col = ColumnInt128::create();
+
+ for (int j = 0; j < row_size; j++) {
+ if (data_set[j].first[i].type() == typeid(Null)) {
+ null_map_data[j] = true;
+ col->insert_default();
+ continue;
+ }
+ auto value = std::any_cast<ut_type::LARGEINT>(data_set[j].first[i]);
+ col->insert_data(reinterpret_cast<char*>(&value), 0);
+ }
+ insert_column_to_block<DataTypeInt128>(columns, ctn, std::move(col),
+ std::move(null_map), block, col_name, i,
+ is_const, row_size);
+ arg_type.type = doris_udf::FunctionContext::TYPE_LARGEINT;
} else if (tp == TypeIndex::Float64) {
auto col = ColumnFloat64::create();
for (int j = 0; j < row_size; j++) {
if (data_set[j].first[i].type() == typeid(Null)) {
null_map_data[j] = true;
- col->insert_value(0);
+ col->insert_default();
continue;
}
- auto value = std::any_cast<double>(data_set[j].first[i]);
+ auto value = std::any_cast<ut_type::DOUBLE>(data_set[j].first[i]);
col->insert_data(reinterpret_cast<char*>(&value), 0);
}
insert_column_to_block<DataTypeFloat64>(columns, ctn, std::move(col),
std::move(null_map), block, col_name, i,
is_const, row_size);
arg_type.type = doris_udf::FunctionContext::TYPE_DOUBLE;
+ } else if (tp == TypeIndex::Decimal128) {
+ auto col = ColumnDecimal<Decimal128>::create(0, 9);
+
+ for (int j = 0; j < row_size; j++) {
+ if (data_set[j].first[i].type() == typeid(Null)) {
+ null_map_data[j] = true;
+ col->insert_default();
+ continue;
+ }
+ auto value = std::any_cast<Decimal<Int128>>(data_set[j].first[i]);
+ col->insert_data(reinterpret_cast<char*>(&value), 0);
+ }
+ insert_column_to_block<DataTypeDecimal<Decimal128>>(columns, ctn, std::move(col),
+ std::move(null_map), block, col_name, i,
+ is_const, row_size);
+ arg_type.type = doris_udf::FunctionContext::TYPE_DECIMALV2;
} else if (tp == TypeIndex::DateTime) {
static std::string date_time_format("%Y-%m-%d %H:%i:%s");
auto col = ColumnInt64::create();
@@ -216,14 +296,15 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
is_const, row_size);
arg_type.type = doris_udf::FunctionContext::TYPE_DATE;
} else {
- DCHECK(false);
+ ASSERT_TRUE(false);
arg_type.type = doris_udf::FunctionContext::INVALID_TYPE;
}
arguments.push_back(i);
arg_types.push_back(arg_type);
if (is_const) {
const auto& column = block.get_by_position(i).column;
- std::shared_ptr<ColumnPtrWrapper> constant_col = std::make_shared<ColumnPtrWrapper>(column);
+ std::shared_ptr<ColumnPtrWrapper> constant_col =
+ std::make_shared<ColumnPtrWrapper>(column);
constant_col_ptrs.push_back(constant_col);
constant_cols.push_back(constant_col.get());
} else {
@@ -235,6 +316,7 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
auto return_type = nullable ? make_nullable(std::make_shared<ReturnType>())
: std::make_shared<ReturnType>();
auto func = SimpleFunctionFactory::instance().get_function(func_name, ctn, return_type);
+ ASSERT_TRUE(func != nullptr);
doris_udf::FunctionContext::TypeDesc fn_ctx_return;
if (std::is_same_v<ReturnType, DataTypeUInt8>) {
@@ -248,7 +330,7 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
} else {
fn_ctx_return.type = doris_udf::FunctionContext::INVALID_TYPE;
}
-
+
FunctionUtils fn_utils(fn_ctx_return, arg_types, 0);
auto* fn_ctx = fn_utils.get_fn_ctx();
fn_ctx->impl()->set_constant_cols(constant_cols);
@@ -256,7 +338,7 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
func->prepare(fn_ctx, FunctionContext::THREAD_LOCAL);
block.insert({nullptr, return_type, "result"});
- DCHECK(func != nullptr);
+
auto result = block.columns() - 1;
func->execute(fn_ctx, block, arguments, result, row_size);
@@ -265,10 +347,11 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
// 3. check the result of function
ColumnPtr column = block.get_columns()[result];
- DCHECK(column != nullptr);
+ ASSERT_TRUE(column != nullptr);
+
for (int i = 0; i < row_size; ++i) {
auto check_column_data = [&]() {
- vectorized::Field field;
+ Field field;
column->get(i, field);
const auto& column_data = field.get<typename ReturnType::FieldType>();
@@ -287,5 +370,5 @@ void check_function(const std::string& func_name, const std::vector<std::any>& i
}
}
}
-} // namespace vectorized
-} // namespace doris
\ No newline at end of file
+
+} // namespace doris::vectorized
diff --git a/be/test/vec/function/function_time_test.cpp b/be/test/vec/function/function_time_test.cpp
index 56152ca..df6747a 100644
--- a/be/test/vec/function/function_time_test.cpp
+++ b/be/test/vec/function/function_time_test.cpp
@@ -28,176 +28,160 @@
#include "runtime/tuple_row.h"
#include "vec/functions/simple_function_factory.h"
-namespace doris {
-
-using vectorized::Null;
-using vectorized::DataSet;
-using vectorized::TypeIndex;
+namespace doris::vectorized {
TEST(TimestampFunctionsTest, day_of_week_test) {
std::string func_name = "dayofweek";
- std::vector<std::any> input_types = {TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2001-02-03 12:34:56")}, 7},
{{std::string("2020-00-01 00:00:00")}, Null()},
{{std::string("2020-01-00 00:00:00")}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, day_of_month_test) {
std::string func_name = "dayofmonth";
- std::vector<std::any> input_types = {TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2020-00-01 00:00:00")}, Null()},
{{std::string("2020-01-01 00:00:00")}, 1},
{{std::string("2020-02-29 00:00:00")}, 29}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, day_of_year_test) {
std::string func_name = "dayofyear";
- std::vector<std::any> input_types = {TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2020-00-01 00:00:00")}, Null()},
{{std::string("2020-01-00 00:00:00")}, Null()},
{{std::string("2020-02-29 00:00:00")}, 60}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, week_of_year_test) {
std::string func_name = "weekofyear";
- std::vector<std::any> input_types = {TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2020-00-01 00:00:00")}, Null()},
{{std::string("2020-01-00 00:00:00")}, Null()},
{{std::string("2020-02-29 00:00:00")}, 9}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, year_test) {
std::string func_name = "year";
- std::vector<std::any> input_types = {TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 00:00:00")}, 2021},
{{std::string("2021-01-00 00:00:00")}, Null()},
{{std::string("2025-05-01 00:00:00")}, 2025}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, quarter_test) {
std::string func_name = "quarter";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 00:00:00")}, 1},
{{std::string("")}, Null()},
{{std::string("2021-01-32 00:00:00")}, Null()},
{{std::string("2025-10-23 00:00:00")}, 4}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, month_test) {
std::string func_name = "month";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 00:00:00")}, 1},
{{std::string("")}, Null()},
{{std::string("2021-01-32 00:00:00")}, Null()},
{{std::string("2025-05-23 00:00:00")}, 5}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, day_test) {
std::string func_name = "day";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 00:00:00")}, 1},
{{std::string("")}, Null()},
{{std::string("2021-01-32 00:00:00")}, Null()},
{{std::string("2025-05-23 00:00:00")}, 23}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, hour_test) {
std::string func_name = "hour";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 23:59:59")}, 23},
{{std::string("2021-01-13 16:56:00")}, 16},
{{std::string("")}, Null()},
{{std::string("2025-05-23 24:00:00")}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, minute_test) {
std::string func_name = "minute";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 23:59:50")}, 59},
{{std::string("2021-01-13 16:20:00")}, 20},
{{std::string("")}, Null()},
{{std::string("2025-05-23 24:00:00")}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, second_test) {
std::string func_name = "second";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 23:50:59")}, 59},
{{std::string("2021-01-13 16:20:00")}, 0},
{{std::string("")}, Null()},
{{std::string("2025-05-23 24:00:00")}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
-//TEST(TimestampFunctionsTest, unix_timestamp_test) {
-// sleep(15);
-// std::string func_name = "unix_timestamp";
-//
-// std::vector<std::any> input_types = {TypeIndex::DateTime};
-//
-// DataSet data_set = {{{std::string("9999-12-30 00:00:00")}, 0},
-// {{std::string("1000-01-01 00:00:00")}, 0}};
-//
-// vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
-//}
-
TEST(TimestampFunctionsTest, from_unix_test) {
std::string func_name = "from_unixtime";
- std::vector<std::any> input_types = {TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32};
DataSet data_set = {{{1565080737}, std::string("2019-08-06 16:38:57")}, {{-123}, Null()}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, timediff_test) {
std::string func_name = "timediff";
- std::vector<std::any> input_types = {TypeIndex::DateTime, TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::DateTime};
DataSet data_set = {
{{std::string("2019-07-18 12:00:00"), std::string("2019-07-18 12:00:00")}, 0.0},
@@ -205,376 +189,311 @@ TEST(TimestampFunctionsTest, timediff_test) {
{{std::string("2019-00-18 12:00:00"), std::string("2019-07-18 13:01:02")}, Null()},
{{std::string("2019-07-18 12:00:00"), std::string("2019-07-00 13:01:02")}, Null()}};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_set);
+ check_function<DataTypeFloat64, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, date_format_test) {
std::string func_name = "date_format";
- std::vector<std::any> input_types = {TypeIndex::DateTime,
- vectorized::Consted {TypeIndex::String}};
+ InputTypeSet input_types = {TypeIndex::DateTime, Consted {TypeIndex::String}};
{
DataSet data_set = {{{std::string("2009-10-04 22:23:00"), std::string("%W %M %Y")},
std::string("Sunday October 2009")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
DataSet data_set = {{{std::string("2007-10-04 22:23:00"), std::string("%H:%i:%s")},
std::string("22:23:00")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
DataSet data_set = {
{{std::string("1900-10-04 22:23:00"), std::string("%D %y %a %d %m %b %j")},
std::string("4th 00 Thu 04 10 Oct 277")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
DataSet data_set = {
{{std::string("1997-10-04 22:23:00"), std::string("%H %k %I %r %T %S %w")},
std::string("22 22 10 10:23:00 PM 22:23:00 00 6")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
DataSet data_set = {{{std::string("1999-01-01 00:00:00"), std::string("%X %V")},
std::string("1998 52")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
DataSet data_set = {
{{std::string("2006-06-01 00:00:00"), std::string("%d")}, std::string("01")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
{
DataSet data_set = {
{{std::string("2006-06-01 00:00:00"), std::string("%%%d")}, std::string("%01")}};
- vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeString, true>(func_name, input_types, data_set);
}
}
TEST(TimestampFunctionsTest, years_add_test) {
std::string func_name = "years_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 00:00:00"), 5},
- vectorized::str_to_data_time("2025-05-23 00:00:00")},
- {{std::string("2020-05-23 00:00:00"), -5},
- vectorized::str_to_data_time("2015-05-23 00:00:00")},
- {{std::string(""), 5}, Null()},
- {{std::string("2020-05-23 00:00:00"), 8000}, Null()},
- {{Null(), 5}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 00:00:00"), 5}, str_to_data_time("2025-05-23 00:00:00")},
+ {{std::string("2020-05-23 00:00:00"), -5}, str_to_data_time("2015-05-23 00:00:00")},
+ {{std::string(""), 5}, Null()},
+ {{std::string("2020-05-23 00:00:00"), 8000}, Null()},
+ {{Null(), 5}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, years_sub_test) {
std::string func_name = "years_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 00:00:00"), 5},
- vectorized::str_to_data_time("2015-05-23 00:00:00")},
- {{std::string("2020-05-23 00:00:00"), -5},
- vectorized::str_to_data_time("2025-05-23 00:00:00")},
- {{std::string(""), 5}, Null()},
- {{std::string("2020-05-23 00:00:00"), 3000}, Null()},
- {{Null(), 5}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 00:00:00"), 5}, str_to_data_time("2015-05-23 00:00:00")},
+ {{std::string("2020-05-23 00:00:00"), -5}, str_to_data_time("2025-05-23 00:00:00")},
+ {{std::string(""), 5}, Null()},
+ {{std::string("2020-05-23 00:00:00"), 3000}, Null()},
+ {{Null(), 5}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, months_add_test) {
std::string func_name = "months_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-10-23 00:00:00"), -4},
- vectorized::str_to_data_time("2020-06-23 00:00:00")},
- {{std::string("2020-05-23 00:00:00"), 4},
- vectorized::str_to_data_time("2020-09-23 00:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 00:00:00"), 10},
- vectorized::str_to_data_time("2021-03-23 00:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-10-23 00:00:00"), -4}, str_to_data_time("2020-06-23 00:00:00")},
+ {{std::string("2020-05-23 00:00:00"), 4}, str_to_data_time("2020-09-23 00:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 00:00:00"), 10}, str_to_data_time("2021-03-23 00:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, months_sub_test) {
std::string func_name = "months_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 00:00:00"), 4},
- vectorized::str_to_data_time("2020-01-23 00:00:00")},
- {{std::string("2020-05-23 00:00:00"), -4},
- vectorized::str_to_data_time("2020-09-23 00:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 00:00:00"), 10},
- vectorized::str_to_data_time("2019-07-23 00:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 00:00:00"), 4}, str_to_data_time("2020-01-23 00:00:00")},
+ {{std::string("2020-05-23 00:00:00"), -4}, str_to_data_time("2020-09-23 00:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 00:00:00"), 10}, str_to_data_time("2019-07-23 00:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, days_add_test) {
std::string func_name = "days_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-10-23 00:00:00"), -4},
- vectorized::str_to_data_time("2020-10-19 00:00:00")},
- {{std::string("2020-05-23 00:00:00"), 4},
- vectorized::str_to_data_time("2020-05-27 00:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 00:00:00"), 10},
- vectorized::str_to_data_time("2020-06-2 00:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-10-23 00:00:00"), -4}, str_to_data_time("2020-10-19 00:00:00")},
+ {{std::string("2020-05-23 00:00:00"), 4}, str_to_data_time("2020-05-27 00:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 00:00:00"), 10}, str_to_data_time("2020-06-2 00:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, days_sub_test) {
std::string func_name = "days_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 00:00:00"), 4},
- vectorized::str_to_data_time("2020-05-19 00:00:00")},
- {{std::string("2020-05-23 00:00:00"), -4},
- vectorized::str_to_data_time("2020-05-27 00:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 00:00:00"), 31},
- vectorized::str_to_data_time("2020-04-22 00:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 00:00:00"), 4}, str_to_data_time("2020-05-19 00:00:00")},
+ {{std::string("2020-05-23 00:00:00"), -4}, str_to_data_time("2020-05-27 00:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 00:00:00"), 31}, str_to_data_time("2020-04-22 00:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, hours_add_test) {
std::string func_name = "hours_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-10-23 10:00:00"), -4},
- vectorized::str_to_data_time("2020-10-23 06:00:00")},
- {{std::string("2020-05-23 10:00:00"), 4},
- vectorized::str_to_data_time("2020-05-23 14:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2020-05-27 14:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-10-23 10:00:00"), -4}, str_to_data_time("2020-10-23 06:00:00")},
+ {{std::string("2020-05-23 10:00:00"), 4}, str_to_data_time("2020-05-23 14:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2020-05-27 14:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, hours_sub_test) {
std::string func_name = "hours_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 10:00:00"), 4},
- vectorized::str_to_data_time("2020-05-23 06:00:00")},
- {{std::string("2020-05-23 10:00:00"), -4},
- vectorized::str_to_data_time("2020-05-23 14:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 31},
- vectorized::str_to_data_time("2020-05-22 03:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 10:00:00"), 4}, str_to_data_time("2020-05-23 06:00:00")},
+ {{std::string("2020-05-23 10:00:00"), -4}, str_to_data_time("2020-05-23 14:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 31}, str_to_data_time("2020-05-22 03:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, minutes_add_test) {
std::string func_name = "minutes_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-10-23 10:00:00"), 40},
- vectorized::str_to_data_time("2020-10-23 10:40:00")},
- {{std::string("2020-05-23 10:00:00"), -40},
- vectorized::str_to_data_time("2020-05-23 09:20:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2020-05-23 11:40:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-10-23 10:00:00"), 40}, str_to_data_time("2020-10-23 10:40:00")},
+ {{std::string("2020-05-23 10:00:00"), -40}, str_to_data_time("2020-05-23 09:20:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2020-05-23 11:40:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, minutes_sub_test) {
std::string func_name = "minutes_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 10:00:00"), 40},
- vectorized::str_to_data_time("2020-05-23 09:20:00")},
- {{std::string("2020-05-23 10:00:00"), -40},
- vectorized::str_to_data_time("2020-05-23 10:40:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2020-05-23 08:20:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 10:00:00"), 40}, str_to_data_time("2020-05-23 09:20:00")},
+ {{std::string("2020-05-23 10:00:00"), -40}, str_to_data_time("2020-05-23 10:40:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2020-05-23 08:20:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, seconds_add_test) {
std::string func_name = "seconds_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-10-23 10:00:00"), 40},
- vectorized::str_to_data_time("2020-10-23 10:00:40")},
- {{std::string("2020-05-23 10:00:00"), -40},
- vectorized::str_to_data_time("2020-05-23 09:59:20")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2020-05-23 10:01:40")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-10-23 10:00:00"), 40}, str_to_data_time("2020-10-23 10:00:40")},
+ {{std::string("2020-05-23 10:00:00"), -40}, str_to_data_time("2020-05-23 09:59:20")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2020-05-23 10:01:40")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, seconds_sub_test) {
std::string func_name = "seconds_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 10:00:00"), 40},
- vectorized::str_to_data_time("2020-05-23 09:59:20")},
- {{std::string("2020-05-23 10:00:00"), -40},
- vectorized::str_to_data_time("2020-05-23 10:00:40")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2020-05-23 09:58:20")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 10:00:00"), 40}, str_to_data_time("2020-05-23 09:59:20")},
+ {{std::string("2020-05-23 10:00:00"), -40}, str_to_data_time("2020-05-23 10:00:40")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2020-05-23 09:58:20")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, weeks_add_test) {
std::string func_name = "weeks_add";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-10-23 10:00:00"), 5},
- vectorized::str_to_data_time("2020-11-27 10:00:00")},
- {{std::string("2020-05-23 10:00:00"), -5},
- vectorized::str_to_data_time("2020-04-18 10:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2022-04-23 10:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-10-23 10:00:00"), 5}, str_to_data_time("2020-11-27 10:00:00")},
+ {{std::string("2020-05-23 10:00:00"), -5}, str_to_data_time("2020-04-18 10:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2022-04-23 10:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, weeks_sub_test) {
std::string func_name = "weeks_sub";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::DateTime, TypeIndex::Int32};
- DataSet data_set = {{{std::string("2020-05-23 10:00:00"), 5},
- vectorized::str_to_data_time("2020-04-18 10:00:00")},
- {{std::string("2020-05-23 10:00:00"), -5},
- vectorized::str_to_data_time("2020-6-27 10:00:00")},
- {{std::string(""), 4}, Null()},
- {{std::string("2020-05-23 10:00:00"), 100},
- vectorized::str_to_data_time("2018-06-23 10:00:00")},
- {{Null(), 4}, Null()}};
+ DataSet data_set = {
+ {{std::string("2020-05-23 10:00:00"), 5}, str_to_data_time("2020-04-18 10:00:00")},
+ {{std::string("2020-05-23 10:00:00"), -5}, str_to_data_time("2020-6-27 10:00:00")},
+ {{std::string(""), 4}, Null()},
+ {{std::string("2020-05-23 10:00:00"), 100}, str_to_data_time("2018-06-23 10:00:00")},
+ {{Null(), 4}, Null()}};
- vectorized::check_function<vectorized::DataTypeDateTime, true>(func_name, input_types,
- data_set);
+ check_function<DataTypeDateTime, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, to_days_test) {
std::string func_name = "to_days";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
DataSet data_set = {{{std::string("2021-01-01 00:00:00")}, 738156},
{{std::string("")}, Null()},
{{std::string("2021-01-32 00:00:00")}, Null()},
{{std::string("0000-01-01 00:00:00")}, 1}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ check_function<DataTypeInt32, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, date_test) {
std::string func_name = "date";
- std::vector<std::any> input_types = {vectorized::TypeIndex::DateTime};
+ InputTypeSet input_types = {TypeIndex::DateTime};
- DataSet data_set = {{{std::string("2021-01-01 06:00:00")},
- vectorized::str_to_data_time("2021-01-01", false)},
- {{std::string("")}, Null()},
- {{Null()}, Null()},
- {{std::string("0000-01-01 00:00:00")},
- vectorized::str_to_data_time("0000-01-01", false)}};
+ DataSet data_set = {
+ {{std::string("2021-01-01 06:00:00")}, str_to_data_time("2021-01-01", false)},
+ {{std::string("")}, Null()},
+ {{Null()}, Null()},
+ {{std::string("0000-01-01 00:00:00")}, str_to_data_time("0000-01-01", false)}};
- vectorized::check_function<vectorized::DataTypeDate, true>(func_name, input_types, data_set);
+ check_function<DataTypeDate, true>(func_name, input_types, data_set);
}
TEST(TimestampFunctionsTest, makedate_test) {
std::string func_name = "makedate";
- std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
- vectorized::TypeIndex::Int32};
+ InputTypeSet input_types = {TypeIndex::Int32, TypeIndex::Int32};
- DataSet data_set = {{{2021, 3}, vectorized::str_to_data_time("2021-01-03", false)},
- {{2021, 95}, vectorized::str_to_data_time("2021-04-05", false)},
- {{2021, 400}, vectorized::str_to_data_time("2022-02-04", false)},
+ DataSet data_set = {{{2021, 3}, str_to_data_time("2021-01-03", false)},
+ {{2021, 95}, str_to_data_time("2021-04-05", false)},
+ {{2021, 400}, str_to_data_time("2022-02-04", false)},
{{2021, 0}, Null()},
{{2021, -10}, Null()},
{{-1, 3}, Null()},
{{12345, 3}, Null()}};
- vectorized::check_function<vectorized::DataTypeDate, true>(func_name, input_types, data_set);
+ check_function<DataTypeDate, true>(func_name, input_types, data_set);
}
-
-} // namespace doris
+} // namespace doris::vectorized
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
diff --git a/run-be-ut.sh b/run-be-ut.sh
index 7af98f7..26a23f7 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -47,6 +47,7 @@ Usage: $0 <options>
--clean clean and build ut
--run build and run all ut
--run xx build and run specified ut
+ -v build and run all vectorized ut
-j build parallel
Eg.
@@ -65,6 +66,7 @@ OPTS=$(getopt \
-l 'run' \
-l 'clean' \
-o 'j:' \
+ -o 'v' \
-- "$@")
if [ $? != 0 ] ; then
@@ -73,20 +75,17 @@ fi
eval set -- "$OPTS"
-PARALLEL=$[$(nproc)/8+1]
-CLEAN=
-RUN=
-if [ $# == 1 ] ; then
- #default
- CLEAN=0
- RUN=0
-else
- CLEAN=0
- RUN=0
+PARALLEL=$[$(nproc)/5+1]
+
+CLEAN=0
+RUN=0
+VECTORIZED_ONLY=0
+if [ $# != 1 ] ; then
while true; do
case "$1" in
--clean) CLEAN=1 ; shift ;;
--run) RUN=1 ; shift ;;
+ -v) VECTORIZED_ONLY=1 ; shift ;;
-j) PARALLEL=$2; shift 2 ;;
--) shift ; break ;;
*) echo "Internal error" ; exit 1 ;;
@@ -168,6 +167,12 @@ cp -r ${DORIS_HOME}/be/test/util/test_data ${DORIS_TEST_BINARY_DIR}/util/
cp -r ${DORIS_HOME}/be/test/plugin/plugin_test ${DORIS_TEST_BINARY_DIR}/plugin/
# find all executable test files
+
+if [ ${VECTORIZED_ONLY} -eq 1 ]; then
+ echo "Run Vectorized ut only"
+ export DORIS_TEST_BINARY_DIR=${DORIS_TEST_BINARY_DIR}/vec
+fi
+
test_files=`find ${DORIS_TEST_BINARY_DIR} -type f -perm -111 -name "*test"`
for test in ${test_files[@]}
@@ -178,5 +183,4 @@ do
$test --gtest_output=xml:${GTEST_OUTPUT_DIR}/${file_name}.xml
fi
done
-
echo "=== Finished. Gtest output: ${GTEST_OUTPUT_DIR}"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 12/33: [Vectorizd] [Function] Add string type vec support at doris_builtins_functions[D (#7661)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 345c119510602110275a698e64f76f7e2f058065
Author: Pxl <95...@qq.com>
AuthorDate: Fri Jan 7 14:57:20 2022 +0800
[Vectorizd] [Function] Add string type vec support at doris_builtins_functions[D (#7661)
---
gensrc/script/doris_builtins_functions.py | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 1399c97..5d96751 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -1045,26 +1045,26 @@ visible_functions = [
'_ZN5doris15StringFunctions17parse_url_prepareEPN9doris_udf'
'15FunctionContextENS2_18FunctionStateScopeE',
'_ZN5doris15StringFunctions15parse_url_closeEPN9doris_udf'
- '15FunctionContextENS2_18FunctionStateScopeE', '', ''],
+ '15FunctionContextENS2_18FunctionStateScopeE', 'vec', ''],
[['parse_url'], 'STRING', ['STRING', 'STRING', 'STRING'],
'_ZN5doris15StringFunctions13parse_url_keyEPN9doris_udf'
'15FunctionContextERKNS1_9StringValES6_S6_',
'_ZN5doris15StringFunctions17parse_url_prepareEPN9doris_udf'
'15FunctionContextENS2_18FunctionStateScopeE',
'_ZN5doris15StringFunctions15parse_url_closeEPN9doris_udf'
- '15FunctionContextENS2_18FunctionStateScopeE', '', ''],
+ '15FunctionContextENS2_18FunctionStateScopeE', 'vec', ''],
[['money_format'], 'STRING', ['BIGINT'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_9BigIntValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['money_format'], 'STRING', ['LARGEINT'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_11LargeIntValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['money_format'], 'STRING', ['DOUBLE'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_9DoubleValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['money_format'], 'STRING', ['DECIMALV2'],
'_ZN5doris15StringFunctions12money_formatEPN9doris_udf15FunctionContextERKNS1_12DecimalV2ValE',
- '', '', '', ''],
+ '', '', 'vec', ''],
[['split_part'], 'STRING', ['STRING', 'STRING', 'INT'],
'_ZN5doris15StringFunctions10split_partEPN9doris_udf15FunctionContextERKNS1_9StringValES6_RKNS1_6IntValE',
'', '', 'vec', 'ALWAYS_NULLABLE'],
@@ -1276,7 +1276,7 @@ visible_functions = [
'15FunctionContextERKNS1_9StringValES6_', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['aes_decrypt'], 'STRING', ['STRING', 'STRING'],
'_ZN5doris19EncryptionFunctions11aes_decryptEPN9doris_udf'
- '15FunctionContextERKNS1_9StringValES6_', '', '', '', ''],
+ '15FunctionContextERKNS1_9StringValES6_', '', '', 'vec', ''],
[['aes_encrypt'], 'STRING', ['STRING', 'STRING', 'STRING', 'STRING'],
'_ZN5doris19EncryptionFunctions11aes_encryptEPN9doris_udf'
'15FunctionContextERKNS1_9StringValES6_S6_S6_', '', '', '', ''],
@@ -1300,7 +1300,7 @@ visible_functions = [
'15FunctionContextERKNS1_9StringValE', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['to_base64'], 'STRING', ['STRING'],
'_ZN5doris19EncryptionFunctions9to_base64EPN9doris_udf'
- '15FunctionContextERKNS1_9StringValE', '', '', '', 'ALWAYS_NULLABLE'],
+ '15FunctionContextERKNS1_9StringValE', '', '', 'vec', 'ALWAYS_NULLABLE'],
[['to_base64'], 'VARCHAR', ['VARCHAR'],
'_ZN5doris19EncryptionFunctions9to_base64EPN9doris_udf'
'15FunctionContextERKNS1_9StringValE', '', '', 'vec', 'ALWAYS_NULLABLE'],
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 02/33: [Bug] Fix negative function error result and sort node eos (#7555)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a33b2ac2d1027e7c3c00c2a0d36276dd1b54df33
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Fri Dec 31 00:37:54 2021 -0600
[Bug] Fix negative function error result and sort node eos (#7555)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/vec/exec/vsort_node.cpp | 1 +
be/src/vec/functions/math.cpp | 9 +--------
be/test/vec/function/function_math_test.cpp | 2 +-
3 files changed, 3 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/exec/vsort_node.cpp b/be/src/vec/exec/vsort_node.cpp
index 79af7c8..734af91 100644
--- a/be/src/vec/exec/vsort_node.cpp
+++ b/be/src/vec/exec/vsort_node.cpp
@@ -84,6 +84,7 @@ Status VSortNode::get_next(RuntimeState* state, Block* block, bool* eos) {
_sorted_blocks[0].skip_num_rows(_offset);
}
block->swap(_sorted_blocks[0]);
+ *eos = true;
} else {
RETURN_IF_ERROR(merge_sort_read(state, block, eos));
}
diff --git a/be/src/vec/functions/math.cpp b/be/src/vec/functions/math.cpp
index af48277..57d6c48 100644
--- a/be/src/vec/functions/math.cpp
+++ b/be/src/vec/functions/math.cpp
@@ -258,14 +258,7 @@ struct NegativeImpl {
using ResultType = A;
static inline ResultType apply(A a) {
- if constexpr (IsDecimalNumber<A>)
- return a > 0 ? A(-a) : a;
- else if constexpr (std::is_integral_v<A> && std::is_signed_v<A>)
- return a > 0 ? static_cast<ResultType>(~a) + 1 : a;
- else if constexpr (std::is_integral_v<A> && std::is_unsigned_v<A>)
- return static_cast<ResultType>(-a);
- else if constexpr (std::is_floating_point_v<A>)
- return static_cast<ResultType>(-std::abs(a));
+ return -a;
}
};
diff --git a/be/test/vec/function/function_math_test.cpp b/be/test/vec/function/function_math_test.cpp
index f56ab7d..0413abd 100644
--- a/be/test/vec/function/function_math_test.cpp
+++ b/be/test/vec/function/function_math_test.cpp
@@ -296,7 +296,7 @@ TEST(MathFunctionTest, negative_test) {
{
std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
- DataSet data_set = {{{0.0123}, -0.0123}, {{90.45}, -90.45}, {{0.0}, 0.0}, {{-60.0}, -60.0}};
+ DataSet data_set = {{{0.0123}, -0.0123}, {{90.45}, -90.45}, {{0.0}, 0.0}, {{-60.0}, 60.0}};
vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
data_set);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 26/33: [Vectorized][Function] Support function stddev/variance/stddev_samp/variance_samp (#7734)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 990723c15b346f0838314e53c8138e574176a5b9
Author: zhangstar333 <87...@users.noreply.github.com>
AuthorDate: Thu Jan 13 20:27:34 2022 +0800
[Vectorized][Function] Support function stddev/variance/stddev_samp/variance_samp (#7734)
---
be/src/exprs/aggregate_functions.cpp | 2 +-
be/src/vec/CMakeLists.txt | 1 +
.../vec/aggregate_functions/aggregate_function.h | 5 +
.../aggregate_functions/aggregate_function_null.h | 9 +-
.../aggregate_function_simple_factory.cpp | 6 +-
.../aggregate_function_simple_factory.h | 20 +-
.../aggregate_function_stddev.cpp | 101 ++++++++
.../aggregate_function_stddev.h | 285 +++++++++++++++++++++
.../java/org/apache/doris/catalog/FunctionSet.java | 68 +++++
9 files changed, 486 insertions(+), 11 deletions(-)
diff --git a/be/src/exprs/aggregate_functions.cpp b/be/src/exprs/aggregate_functions.cpp
index a22c3ee..93166cf 100644
--- a/be/src/exprs/aggregate_functions.cpp
+++ b/be/src/exprs/aggregate_functions.cpp
@@ -1874,8 +1874,8 @@ static double compute_knuth_variance(const KnuthVarianceState& state, bool pop)
static DecimalV2Value decimalv2_compute_knuth_variance(const DecimalV2KnuthVarianceState& state,
bool pop) {
DecimalV2Value new_count = DecimalV2Value();
- new_count.assign_from_double(state.count);
if (state.count == 1) return new_count;
+ new_count.assign_from_double(state.count);
DecimalV2Value new_m2 = DecimalV2Value::from_decimal_val(state.m2);
if (pop)
return new_m2 / new_count;
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index aa302ce..9c4d947 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -31,6 +31,7 @@ set(VEC_FILES
aggregate_functions/aggregate_function_bitmap.cpp
aggregate_functions/aggregate_function_reader.cpp
aggregate_functions/aggregate_function_window.cpp
+ aggregate_functions/aggregate_function_stddev.cpp
aggregate_functions/aggregate_function_simple_factory.cpp
columns/collator.cpp
columns/column.cpp
diff --git a/be/src/vec/aggregate_functions/aggregate_function.h b/be/src/vec/aggregate_functions/aggregate_function.h
index 412382f..4c2ef36 100644
--- a/be/src/vec/aggregate_functions/aggregate_function.h
+++ b/be/src/vec/aggregate_functions/aggregate_function.h
@@ -114,6 +114,11 @@ public:
*/
virtual bool is_state() const { return false; }
+ /// if return false, during insert_result_into function, you colud get nullable result column,
+ /// so could insert to null value by yourself, rather than by AggregateFunctionNullBase;
+ /// because you maybe be calculate a invalid value, but want to use null replace it;
+ virtual bool insert_to_null_default() const { return true; }
+
/** The inner loop that uses the function pointer is better than using the virtual function.
* The reason is that in the case of virtual functions GCC 5.1.2 generates code,
* which, at each iteration of the loop, reloads the function address (the offset value in the virtual function table) from memory to the register.
diff --git a/be/src/vec/aggregate_functions/aggregate_function_null.h b/be/src/vec/aggregate_functions/aggregate_function_null.h
index 4c61e2b..9458d7d 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_null.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_null.h
@@ -144,9 +144,12 @@ public:
if constexpr (result_is_nullable) {
ColumnNullable& to_concrete = assert_cast<ColumnNullable&>(to);
if (get_flag(place)) {
- nested_function->insert_result_into(nested_place(place),
- to_concrete.get_nested_column());
- to_concrete.get_null_map_data().push_back(0);
+ if (nested_function->insert_to_null_default()) {
+ nested_function->insert_result_into(nested_place(place), to_concrete.get_nested_column());
+ to_concrete.get_null_map_data().push_back(0);
+ } else {
+ nested_function->insert_result_into(nested_place(place), to); //want to insert into null value by self
+ }
} else {
to_concrete.insert_default();
}
diff --git a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
index 8a1995b..ba1b2ba 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
+++ b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
@@ -35,6 +35,7 @@ void register_aggregate_function_combinator_distinct(AggregateFunctionSimpleFact
void register_aggregate_function_bitmap(AggregateFunctionSimpleFactory& factory);
void register_aggregate_function_window_rank(AggregateFunctionSimpleFactory& factory);
void register_aggregate_function_window_lead_lag(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_stddev_variance(AggregateFunctionSimpleFactory& factory);
AggregateFunctionSimpleFactory& AggregateFunctionSimpleFactory::instance() {
static std::once_flag oc;
static AggregateFunctionSimpleFactory instance;
@@ -49,10 +50,11 @@ AggregateFunctionSimpleFactory& AggregateFunctionSimpleFactory::instance() {
register_aggregate_function_HLL_union_agg(instance);
register_aggregate_function_reader(instance); // register aggregate function for agg reader
register_aggregate_function_window_rank(instance);
-
+ register_aggregate_function_stddev_variance(instance);
+
// if you only register function with no nullable, and wants to add nullable automatically, you should place function above this line
register_aggregate_function_combinator_null(instance);
-
+
register_aggregate_function_reader_no_spread(instance);
register_aggregate_function_window_lead_lag(instance);
});
diff --git a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h
index d12c27a..1bac4f1 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.h
@@ -45,7 +45,7 @@ private:
AggregateFunctions aggregate_functions;
AggregateFunctions nullable_aggregate_functions;
-
+ std::unordered_map<std::string, std::string> function_alias;
public:
void register_nullable_function_combinator(const Creator& creator) {
for (const auto& entity : aggregate_functions) {
@@ -77,15 +77,21 @@ public:
nullable = true;
}
}
+
+ std::string name_str = name;
+ if (function_alias.count(name)) {
+ name_str = function_alias[name];
+ }
+
if (nullable) {
- return nullable_aggregate_functions.find(name) == nullable_aggregate_functions.end()
+ return nullable_aggregate_functions.find(name_str) == nullable_aggregate_functions.end()
? nullptr
- : nullable_aggregate_functions[name](name, argument_types, parameters,
+ : nullable_aggregate_functions[name_str](name_str, argument_types, parameters,
result_is_nullable);
} else {
- return aggregate_functions.find(name) == aggregate_functions.end()
+ return aggregate_functions.find(name_str) == aggregate_functions.end()
? nullptr
- : aggregate_functions[name](name, argument_types, parameters,
+ : aggregate_functions[name_str](name_str, argument_types, parameters,
result_is_nullable);
}
}
@@ -98,6 +104,10 @@ public:
}
}
+ void register_alias(const std::string& name, const std::string& alias) {
+ function_alias[alias] = name;
+ }
+
public:
static AggregateFunctionSimpleFactory& instance();
};
diff --git a/be/src/vec/aggregate_functions/aggregate_function_stddev.cpp b/be/src/vec/aggregate_functions/aggregate_function_stddev.cpp
new file mode 100644
index 0000000..f1794d6
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_stddev.cpp
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/aggregate_functions/aggregate_function_stddev.h"
+
+#include "common/logging.h"
+#include "vec/aggregate_functions/aggregate_function_simple_factory.h"
+#include "vec/aggregate_functions/factory_helpers.h"
+#include "vec/aggregate_functions/helpers.h"
+namespace doris::vectorized {
+
+template <template <typename> class AggregateFunctionTemplate, template <typename> class NameData,
+ template <typename, typename> class Data, bool is_stddev>
+static IAggregateFunction* create_function_single_value(const String& name,
+ const DataTypes& argument_types,
+ const Array& parameters) {
+ auto type = argument_types[0].get();
+ if (type->is_nullable()) {
+ type = assert_cast<const DataTypeNullable*>(type)->get_nested_type().get();
+ }
+ WhichDataType which(*type);
+
+#define DISPATCH(TYPE) \
+ if (which.idx == TypeIndex::TYPE) \
+ return new AggregateFunctionTemplate<NameData<Data<TYPE, BaseData<TYPE, is_stddev>>>>( \
+ argument_types);
+ FOR_NUMERIC_TYPES(DISPATCH)
+#undef DISPATCH
+ if (which.is_decimal()) {
+ return new AggregateFunctionTemplate<
+ NameData<Data<Decimal128, BaseDatadecimal<is_stddev>>>>(argument_types);
+ }
+ DCHECK(false) << "with unknowed type, failed in create_aggregate_function_stddev_variance";
+ return nullptr;
+}
+
+template <bool is_stddev>
+AggregateFunctionPtr create_aggregate_function_variance_samp(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<AggregateFunctionStddevSamp, VarianceSampData, SampData,
+ is_stddev>(name, argument_types, parameters));
+}
+
+template <bool is_stddev>
+AggregateFunctionPtr create_aggregate_function_stddev_samp(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<AggregateFunctionStddevSamp, StddevSampData, SampData,
+ is_stddev>(name, argument_types, parameters));
+}
+
+template <bool is_stddev>
+AggregateFunctionPtr create_aggregate_function_variance_pop(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<AggregateFunctionStddevSamp, VarianceData, PopData,
+ is_stddev>(name, argument_types, parameters));
+}
+
+template <bool is_stddev>
+AggregateFunctionPtr create_aggregate_function_stddev_pop(const std::string& name,
+ const DataTypes& argument_types,
+ const Array& parameters,
+ const bool result_is_nullable) {
+ return AggregateFunctionPtr(
+ create_function_single_value<AggregateFunctionStddevSamp, StddevData, PopData,
+ is_stddev>(name, argument_types, parameters));
+}
+
+void register_aggregate_function_stddev_variance(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function("variance_samp", create_aggregate_function_variance_samp<false>);
+ factory.register_function("variance", create_aggregate_function_variance_pop<false>);
+ factory.register_alias("variance_samp", "var_samp");
+ factory.register_alias("variance", "var_pop");
+ factory.register_alias("variance", "variance_pop");
+ factory.register_function("stddev_samp", create_aggregate_function_stddev_samp<true>);
+ factory.register_function("stddev", create_aggregate_function_stddev_pop<true>);
+ factory.register_alias("stddev", "stddev_pop");
+}
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/aggregate_functions/aggregate_function_stddev.h b/be/src/vec/aggregate_functions/aggregate_function_stddev.h
new file mode 100644
index 0000000..6cdba200
--- /dev/null
+++ b/be/src/vec/aggregate_functions/aggregate_function_stddev.h
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "common/status.h"
+#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/io/io_helper.h"
+namespace doris::vectorized {
+
+template <typename T, bool is_stddev>
+struct BaseData {
+ BaseData() : mean(0.0), m2(0.0), count(0) {}
+
+ void write(BufferWritable& buf) const {
+ write_binary(mean, buf);
+ write_binary(m2, buf);
+ write_binary(count, buf);
+ }
+
+ void read(BufferReadable& buf) {
+ read_binary(mean, buf);
+ read_binary(m2, buf);
+ read_binary(count, buf);
+ }
+
+ void reset() {
+ mean = 0.0;
+ m2 = 0.0;
+ count = 0;
+ }
+
+ double get_result(double res) const {
+ if constexpr (is_stddev) {
+ return std::sqrt(res);
+ } else {
+ return res;
+ }
+ }
+
+ double get_pop_result() const {
+ if (count == 1) {
+ return 0.0;
+ }
+ double res = m2 / count;
+ return get_result(res);
+ }
+
+ double get_samp_result() const {
+ double res = m2 / (count - 1);
+ return get_result(res);
+ }
+
+ static const DataTypePtr get_return_type() {
+ return make_nullable(std::make_shared<DataTypeNumber<Float64>>());
+ }
+
+ void merge(const BaseData& rhs) {
+ if (rhs.count == 0) {
+ return;
+ }
+ double delta = mean - rhs.mean;
+ double sum_count = count + rhs.count;
+ mean = rhs.mean + delta * count / sum_count;
+ m2 = rhs.m2 + m2 + (delta * delta) * rhs.count * count / sum_count;
+ count = sum_count;
+ }
+
+ void add(const IColumn** columns, size_t row_num) {
+ const auto& sources = static_cast<const ColumnVector<T>&>(*columns[0]);
+ double source_data = sources.get_data()[row_num];
+
+ double delta = source_data - mean;
+ double r = delta / (1 + count);
+ mean += r;
+ m2 += count * delta * r;
+ count += 1;
+ }
+
+ double mean;
+ double m2;
+ int64_t count;
+};
+
+template <bool is_stddev>
+struct BaseDatadecimal {
+ BaseDatadecimal() : mean(0), m2(0), count(0) {}
+
+ void write(BufferWritable& buf) const {
+ write_binary(mean, buf);
+ write_binary(m2, buf);
+ write_binary(count, buf);
+ }
+
+ void read(BufferReadable& buf) {
+ read_binary(mean, buf);
+ read_binary(m2, buf);
+ read_binary(count, buf);
+ }
+
+ void reset() {
+ mean = DecimalV2Value();
+ m2 = DecimalV2Value();
+ count = {};
+ }
+
+ DecimalV2Value get_result(DecimalV2Value res) const {
+ if constexpr (is_stddev) {
+ return DecimalV2Value::sqrt(res);
+ } else {
+ return res;
+ }
+ }
+
+ DecimalV2Value get_pop_result() const {
+ DecimalV2Value new_count = DecimalV2Value();
+ if (count == 1) {
+ return new_count;
+ }
+ DecimalV2Value res = m2 / new_count.assign_from_double(count);
+ return get_result(res);
+ }
+
+ DecimalV2Value get_samp_result() const {
+ DecimalV2Value new_count = DecimalV2Value();
+ DecimalV2Value res = m2 / new_count.assign_from_double(count - 1);
+ return get_result(res);
+ }
+
+ static const DataTypePtr get_return_type() {
+ return make_nullable(std::make_shared<DataTypeDecimal<Decimal128>>(27, 9));
+ }
+
+ void merge(const BaseDatadecimal& rhs) {
+ if (rhs.count == 0) {
+ return;
+ }
+ DecimalV2Value new_count = DecimalV2Value();
+ new_count.assign_from_double(count);
+ DecimalV2Value rhs_count = DecimalV2Value();
+ rhs_count.assign_from_double(rhs.count);
+
+ DecimalV2Value delta = mean - rhs.mean;
+ DecimalV2Value sum_count = new_count + rhs_count;
+ mean = rhs.mean + delta * (new_count / sum_count);
+ m2 = rhs.m2 + m2 + (delta * delta) * (rhs_count * new_count / sum_count);
+ count += rhs.count;
+ }
+
+ void add(const IColumn** columns, size_t row_num) {
+ DecimalV2Value source_data = DecimalV2Value();
+ const auto& sources = static_cast<const ColumnDecimal<Decimal128>&>(*columns[0]);
+ source_data = (DecimalV2Value)sources.get_data()[row_num];
+
+ DecimalV2Value new_count = DecimalV2Value();
+ new_count.assign_from_double(count);
+ DecimalV2Value increase_count = DecimalV2Value();
+ increase_count.assign_from_double(1 + count);
+
+ DecimalV2Value delta = source_data - mean;
+ DecimalV2Value r = delta / increase_count;
+ mean += r;
+ m2 += new_count * delta * r;
+ count += 1;
+ }
+
+ DecimalV2Value mean;
+ DecimalV2Value m2;
+ int64_t count;
+};
+
+template <typename T, typename Data>
+struct PopData : Data {
+ using ColVecResult = std::conditional_t<IsDecimalNumber<T>, ColumnDecimal<Decimal128>, ColumnVector<Float64>>;
+ void insert_result_into(IColumn& to) const {
+ ColumnNullable& nullable_column = assert_cast<ColumnNullable&>(to);
+ auto& col = static_cast<ColVecResult&>(nullable_column.get_nested_column());
+ if constexpr (IsDecimalNumber<T>) {
+ col.get_data().push_back(this->get_pop_result().value());
+ } else {
+ col.get_data().push_back(this->get_pop_result());
+ }
+ nullable_column.get_null_map_data().push_back(0);
+ }
+};
+
+template <typename T, typename Data>
+struct SampData : Data {
+ using ColVecResult = std::conditional_t<IsDecimalNumber<T>, ColumnDecimal<Decimal128>, ColumnVector<Float64>>;
+ void insert_result_into(IColumn& to) const {
+ ColumnNullable& nullable_column = assert_cast<ColumnNullable&>(to);
+ if (this->count == 1) {
+ nullable_column.insert_default();
+ } else {
+ auto& col = static_cast<ColVecResult&>(nullable_column.get_nested_column());
+ if constexpr (IsDecimalNumber<T>) {
+ col.get_data().push_back(this->get_samp_result().value());
+ } else {
+ col.get_data().push_back(this->get_samp_result());
+ }
+ nullable_column.get_null_map_data().push_back(0);
+ }
+ }
+};
+
+template <typename Data>
+struct StddevData : Data {
+ static const char* name() { return "stddev"; }
+};
+
+template <typename Data>
+struct VarianceData : Data {
+ static const char* name() { return "variance"; }
+};
+
+template <typename Data>
+struct VarianceSampData : Data {
+ static const char* name() { return "variance_samp"; }
+};
+
+template <typename Data>
+struct StddevSampData : Data {
+ static const char* name() { return "stddev_samp"; }
+};
+
+template <typename Data>
+class AggregateFunctionStddevSamp final
+ : public IAggregateFunctionDataHelper<Data, AggregateFunctionStddevSamp<Data>> {
+public:
+ AggregateFunctionStddevSamp(const DataTypes& argument_types_)
+ : IAggregateFunctionDataHelper<Data, AggregateFunctionStddevSamp<Data>>(argument_types_,
+ {}) {}
+
+ String get_name() const override { return Data::name(); }
+
+ bool insert_to_null_default() const override { return false; }
+
+ DataTypePtr get_return_type() const override { return Data::get_return_type(); }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, size_t row_num,
+ Arena*) const override {
+ this->data(place).add(columns, row_num);
+ }
+
+ void reset(AggregateDataPtr __restrict place) const override { this->data(place).reset(); }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena*) const override {
+ this->data(place).merge(this->data(rhs));
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ this->data(place).write(buf);
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena*) const override {
+ this->data(place).read(buf);
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ this->data(place).insert_result_into(to);
+ }
+
+ const char* get_header_file_path() const override { return __FILE__; }
+};
+
+} // namespace doris::vectorized
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
index 7c0cde1..0c8079c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
@@ -1674,6 +1674,74 @@ public class FunctionSet<min_initIN9doris_udf12DecimalV2ValEEEvPNS2_15FunctionCo
null,
prefix + STDDEV_POP_FINALIZE_SYMBOL.get(t),
false, false, false));
+ //vec stddev stddev_samp stddev_pop
+ addBuiltin(AggregateFunction.createBuiltin("stddev",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + STDDEV_POP_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("stddev_samp",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + STDDEV_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("stddev_pop",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + STDDEV_POP_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+
+ //vec: variance variance_samp var_samp variance_pop var_pop
+ addBuiltin(AggregateFunction.createBuiltin("variance",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + VAR_POP_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("variance_pop",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + VAR_POP_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("var_pop",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + VAR_POP_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("variance_samp",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + VAR_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("var_samp",
+ Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), t,
+ prefix + STDDEV_INIT_SYMBOL.get(t),
+ prefix + STDDEV_UPDATE_SYMBOL.get(t),
+ prefix + STDDEV_MERGE_SYMBOL.get(t),
+ null,
+ prefix + VAR_FINALIZE_SYMBOL.get(t),
+ false, false, false, true));
+
addBuiltin(AggregateFunction.createBuiltin("variance",
Lists.newArrayList(t), STDDEV_RETTYPE_SYMBOL.get(t), Type.VARCHAR,
prefix + STDDEV_INIT_SYMBOL.get(t),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 31/33: [Vectorized](improving) (exec) optimize VDataStreamSender's send() performance #7747 (#7751)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit f45d2a2144179bcc64b807a68f4a6befdc796383
Author: zuochunwei <zc...@qq.com>
AuthorDate: Mon Jan 17 17:10:24 2022 +0800
[Vectorized](improving) (exec) optimize VDataStreamSender's send() performance #7747 (#7751)
---
be/src/vec/columns/column.h | 4 ++
be/src/vec/columns/column_complex.h | 10 ++++-
be/src/vec/columns/column_const.h | 4 ++
be/src/vec/columns/column_decimal.h | 10 +++++
be/src/vec/columns/column_dummy.h | 4 ++
be/src/vec/columns/column_nullable.cpp | 6 +++
be/src/vec/columns/column_nullable.h | 1 +
be/src/vec/columns/column_string.cpp | 6 +++
be/src/vec/columns/column_string.h | 2 +
be/src/vec/columns/column_vector.cpp | 10 +++++
be/src/vec/columns/column_vector.h | 2 +
be/src/vec/columns/predicate_column.h | 6 ++-
be/src/vec/core/block.cpp | 13 +++++-
be/src/vec/core/block.h | 2 +
be/src/vec/sink/vdata_stream_sender.cpp | 79 ++++++++++++++++++++++-----------
be/src/vec/sink/vdata_stream_sender.h | 36 ++++++++++++++-
16 files changed, 164 insertions(+), 31 deletions(-)
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index a869a65..d58979d 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -160,6 +160,10 @@ public:
virtual void insert_many_from(const IColumn& src, size_t position, size_t length) {
for (size_t i = 0; i < length; ++i) insert_from(src, position);
}
+
+ /// Appends a batch elements from other column with the same type
+ /// indices_begin + indices_end represent the row indices of column src
+ virtual void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) = 0;
/// Appends data located in specified memory chunk if it is possible (throws an exception if it cannot be implemented).
/// Is used to optimize some computations (in aggregation, for example).
diff --git a/be/src/vec/columns/column_complex.h b/be/src/vec/columns/column_complex.h
index 296f94b..18794d3 100644
--- a/be/src/vec/columns/column_complex.h
+++ b/be/src/vec/columns/column_complex.h
@@ -127,6 +127,14 @@ public:
data.insert(data.end(), st, ed);
}
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override {
+ const Self& src_vec = assert_cast<const Self&>(src);
+ data.reserve(size() + (indices_end - indices_begin));
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ data.push_back(src_vec.get_element(*x));
+ }
+ }
+
void pop_back(size_t n) { data.erase(data.end() - n, data.end()); }
// it's impossable to use ComplexType as key , so we don't have to implemnt them
[[noreturn]] StringRef serialize_value_into_arena(size_t n, Arena& arena,
@@ -286,4 +294,4 @@ ColumnPtr ColumnComplexType<T>::replicate(const IColumn::Offsets& offsets) const
}
using ColumnBitmap = ColumnComplexType<BitmapValue>;
-} // namespace doris::vectorized
\ No newline at end of file
+} // namespace doris::vectorized
diff --git a/be/src/vec/columns/column_const.h b/be/src/vec/columns/column_const.h
index 703e226..e019c56 100644
--- a/be/src/vec/columns/column_const.h
+++ b/be/src/vec/columns/column_const.h
@@ -84,6 +84,10 @@ public:
s += length;
}
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override {
+ s += (indices_end - indices_begin);
+ }
+
void insert(const Field&) override { ++s; }
void insert_data(const char*, size_t) override { ++s; }
diff --git a/be/src/vec/columns/column_decimal.h b/be/src/vec/columns/column_decimal.h
index 46412cb..67f4fa9 100644
--- a/be/src/vec/columns/column_decimal.h
+++ b/be/src/vec/columns/column_decimal.h
@@ -26,6 +26,7 @@
#include "vec/columns/column_impl.h"
#include "vec/columns/column_vector_helper.h"
#include "vec/common/typeid_cast.h"
+#include "vec/common/assert_cast.h"
#include "vec/core/field.h"
namespace doris::vectorized {
@@ -95,6 +96,15 @@ public:
void insert_from(const IColumn& src, size_t n) override {
data.push_back(static_cast<const Self&>(src).get_data()[n]);
}
+
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override {
+ const Self& src_vec = assert_cast<const Self&>(src);
+ data.reserve(size() + (indices_end - indices_begin));
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ data.push_back_without_reserve(src_vec.get_element(*x));
+ }
+ }
+
void insert_data(const char* pos, size_t /*length*/) override;
void insert_default() override { data.push_back(T()); }
void insert(const Field& x) override {
diff --git a/be/src/vec/columns/column_dummy.h b/be/src/vec/columns/column_dummy.h
index 4531058..cac3a06 100644
--- a/be/src/vec/columns/column_dummy.h
+++ b/be/src/vec/columns/column_dummy.h
@@ -80,6 +80,10 @@ public:
s += length;
}
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override {
+ s += (indices_end - indices_begin);
+ }
+
ColumnPtr filter(const Filter& filt, ssize_t /*result_size_hint*/) const override {
return clone_dummy(count_bytes_in_filter(filt));
}
diff --git a/be/src/vec/columns/column_nullable.cpp b/be/src/vec/columns/column_nullable.cpp
index bf4bb44..ae3a2fd 100644
--- a/be/src/vec/columns/column_nullable.cpp
+++ b/be/src/vec/columns/column_nullable.cpp
@@ -131,6 +131,12 @@ void ColumnNullable::insert_range_from(const IColumn& src, size_t start, size_t
get_nested_column().insert_range_from(*nullable_col.nested_column, start, length);
}
+void ColumnNullable::insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) {
+ const ColumnNullable& src_concrete = assert_cast<const ColumnNullable&>(src);
+ get_nested_column().insert_indices_from(src_concrete.get_nested_column(), indices_begin, indices_end);
+ get_null_map_column().insert_indices_from(src_concrete.get_null_map_column(), indices_begin, indices_end);
+}
+
void ColumnNullable::insert(const Field& x) {
if (x.is_null()) {
get_nested_column().insert_default();
diff --git a/be/src/vec/columns/column_nullable.h b/be/src/vec/columns/column_nullable.h
index fb6aa8f..8163149 100644
--- a/be/src/vec/columns/column_nullable.h
+++ b/be/src/vec/columns/column_nullable.h
@@ -83,6 +83,7 @@ public:
StringRef serialize_value_into_arena(size_t n, Arena& arena, char const*& begin) const override;
const char* deserialize_and_insert_from_arena(const char* pos) override;
void insert_range_from(const IColumn& src, size_t start, size_t length) override;
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override;
void insert(const Field& x) override;
void insert_from(const IColumn& src, size_t n) override;
diff --git a/be/src/vec/columns/column_string.cpp b/be/src/vec/columns/column_string.cpp
index 05a702b..f4a32dc 100644
--- a/be/src/vec/columns/column_string.cpp
+++ b/be/src/vec/columns/column_string.cpp
@@ -93,6 +93,12 @@ void ColumnString::insert_range_from(const IColumn& src, size_t start, size_t le
}
}
+void ColumnString::insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) {
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ ColumnString::insert_from(src, *x);
+ }
+}
+
ColumnPtr ColumnString::filter(const Filter& filt, ssize_t result_size_hint) const {
if (offsets.size() == 0) return ColumnString::create();
diff --git a/be/src/vec/columns/column_string.h b/be/src/vec/columns/column_string.h
index 44a6608..2234490 100644
--- a/be/src/vec/columns/column_string.h
+++ b/be/src/vec/columns/column_string.h
@@ -187,6 +187,8 @@ public:
void insert_range_from(const IColumn& src, size_t start, size_t length) override;
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override;
+
ColumnPtr filter(const Filter& filt, ssize_t result_size_hint) const override;
ColumnPtr permute(const Permutation& perm, size_t limit) const override;
diff --git a/be/src/vec/columns/column_vector.cpp b/be/src/vec/columns/column_vector.cpp
index 017ae29..75ff144 100644
--- a/be/src/vec/columns/column_vector.cpp
+++ b/be/src/vec/columns/column_vector.cpp
@@ -29,6 +29,7 @@
#include "runtime/datetime_value.h"
#include "vec/columns/columns_common.h"
#include "vec/common/arena.h"
+#include "vec/common/assert_cast.h"
#include "vec/common/bit_cast.h"
#include "vec/common/exception.h"
#include "vec/common/nan_utils.h"
@@ -218,6 +219,15 @@ void ColumnVector<T>::insert_range_from(const IColumn& src, size_t start, size_t
}
template <typename T>
+void ColumnVector<T>::insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) {
+ const Self& src_vec = assert_cast<const Self&>(src);
+ data.reserve(size() + (indices_end - indices_begin));
+ for (auto x = indices_begin; x != indices_end; ++x) {
+ data.push_back_without_reserve(src_vec.get_element(*x));
+ }
+}
+
+template <typename T>
ColumnPtr ColumnVector<T>::filter(const IColumn::Filter& filt, ssize_t result_size_hint) const {
size_t size = data.size();
if (size != filt.size()) {
diff --git a/be/src/vec/columns/column_vector.h b/be/src/vec/columns/column_vector.h
index 6626229..2c5647e 100644
--- a/be/src/vec/columns/column_vector.h
+++ b/be/src/vec/columns/column_vector.h
@@ -200,6 +200,8 @@ public:
void insert_range_from(const IColumn& src, size_t start, size_t length) override;
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override;
+
ColumnPtr filter(const IColumn::Filter& filt, ssize_t result_size_hint) const override;
// note(wb) this method is only used in storage layer now
diff --git a/be/src/vec/columns/predicate_column.h b/be/src/vec/columns/predicate_column.h
index 8095cff..a23c550 100644
--- a/be/src/vec/columns/predicate_column.h
+++ b/be/src/vec/columns/predicate_column.h
@@ -148,6 +148,10 @@ public:
LOG(FATAL) << "insert_range_from not supported in PredicateColumnType";
}
+ void insert_indices_from(const IColumn& src, const int* indices_begin, const int* indices_end) override {
+ LOG(FATAL) << "insert_indices_from not supported in PredicateColumnType";
+ }
+
void pop_back(size_t n) override {
LOG(FATAL) << "pop_back not supported in PredicateColumnType";
}
@@ -440,4 +444,4 @@ private:
};
using ColumnStringValue = PredicateColumnType<StringValue>;
-} // namespace
\ No newline at end of file
+} // namespace
diff --git a/be/src/vec/core/block.cpp b/be/src/vec/core/block.cpp
index d200a46..3664a2a 100644
--- a/be/src/vec/core/block.cpp
+++ b/be/src/vec/core/block.cpp
@@ -837,9 +837,18 @@ size_t MutableBlock::rows() const {
}
void MutableBlock::add_row(const Block* block, int row) {
- auto& src_columns_with_schema = block->get_columns_with_type_and_name();
+ auto& block_data = block->get_columns_with_type_and_name();
for (size_t i = 0; i < _columns.size(); ++i) {
- _columns[i]->insert_from(*src_columns_with_schema[i].column.get(), row);
+ _columns[i]->insert_from(*block_data[i].column.get(), row);
+ }
+}
+
+void MutableBlock::add_rows(const Block* block, const int* row_begin, const int* row_end) {
+ auto& block_data = block->get_columns_with_type_and_name();
+ for (size_t i = 0; i < _columns.size(); ++i) {
+ auto& dst = _columns[i];
+ auto& src = *block_data[i].column.get();
+ dst->insert_indices_from(src, row_begin, row_end);
}
}
diff --git a/be/src/vec/core/block.h b/be/src/vec/core/block.h
index 1c435ee..a39baa9 100644
--- a/be/src/vec/core/block.h
+++ b/be/src/vec/core/block.h
@@ -339,6 +339,8 @@ public:
Block to_block(int start_column, int end_column);
void add_row(const Block* block, int row);
+ void add_rows(const Block* block, const int* row_begin, const int* row_end);
+
std::string dump_data(size_t row_limit = 100) const;
void clear() {
diff --git a/be/src/vec/sink/vdata_stream_sender.cpp b/be/src/vec/sink/vdata_stream_sender.cpp
index 0cc5c66..d953c9d 100644
--- a/be/src/vec/sink/vdata_stream_sender.cpp
+++ b/be/src/vec/sink/vdata_stream_sender.cpp
@@ -169,6 +169,42 @@ Status VDataStreamSender::Channel::add_row(Block* block, int row) {
return Status::OK();
}
+Status VDataStreamSender::Channel::add_rows(Block* block, const std::vector<int>& rows) {
+ if (_fragment_instance_id.lo == -1) {
+ return Status::OK();
+ }
+
+ if (_mutable_block.get() == nullptr) {
+ auto empty_block = block->clone_empty();
+ _mutable_block.reset(
+ new MutableBlock(empty_block.mutate_columns(), empty_block.get_data_types()));
+ }
+
+ int row_wait_add = rows.size();
+ int batch_size = _parent->state()->batch_size();
+ const int* begin = &rows[0];
+
+ while (row_wait_add > 0) {
+ int row_add, max_add = batch_size - _mutable_block->rows();
+ if (row_wait_add >= max_add) {
+ row_add = max_add;
+ } else {
+ row_add = row_wait_add;
+ }
+
+ _mutable_block->add_rows(block, begin, begin + row_add);
+
+ row_wait_add -= row_add;
+ begin += row_add;
+
+ if (row_add == max_add) {
+ RETURN_IF_ERROR(send_current_block());
+ }
+ }
+
+ return Status::OK();
+}
+
Status VDataStreamSender::Channel::close_wait(RuntimeState* state) {
if (_need_close) {
Status st = _wait_last_brpc();
@@ -394,51 +430,49 @@ Status VDataStreamSender::send(RuntimeState* state, Block* block) {
int num_channels = _channels.size();
// will only copy schema
// we don't want send temp columns
- auto send_block = *block;
- std::vector<int> result(_partition_expr_ctxs.size());
- int counter = 0;
+ int result_size = _partition_expr_ctxs.size();
+ int result[result_size];
+ RETURN_IF_ERROR(get_partition_column_result(block, result));
- for (auto ctx : _partition_expr_ctxs) {
- RETURN_IF_ERROR(ctx->execute(block, &result[counter++]));
- }
// vectorized caculate hash
int rows = block->rows();
// for each row, we have a siphash val
std::vector<SipHash> siphashs(rows);
// result[j] means column index, i means rows index
- for (int j = 0; j < result.size(); ++j) {
+ for (int j = 0; j < result_size; ++j) {
auto column = block->get_by_position(result[j]).column;
for (int i = 0; i < rows; ++i) {
column->update_hash_with_value(i, siphashs[i]);
}
}
+ // channel2rows' subscript means channel id
+ std::vector<vectorized::UInt64> hash_vals(rows);
for (int i = 0; i < rows; i++) {
- auto target_channel_id = siphashs[i].get64() % num_channels;
- RETURN_IF_ERROR(_channels[target_channel_id]->add_row(&send_block, i));
+ hash_vals[i] = siphashs[i].get64();
}
+ RETURN_IF_ERROR(channel_add_rows(_channels, num_channels, hash_vals, rows, block));
} else if (_part_type == TPartitionType::BUCKET_SHFFULE_HASH_PARTITIONED) {
// 1. caculate hash
// 2. dispatch rows to channel
int num_channels = _channel_shared_ptrs.size();
- auto send_block = *block;
- std::vector<int> result(_partition_expr_ctxs.size());
- int counter = 0;
- for (auto ctx : _partition_expr_ctxs) {
- RETURN_IF_ERROR(ctx->execute(block, &result[counter++]));
- }
+
+ int result_size = _partition_expr_ctxs.size();
+ int result[result_size];
+ RETURN_IF_ERROR(get_partition_column_result(block, result));
+
// vectorized caculate hash val
int rows = block->rows();
// for each row, we have a hash_val
std::vector<size_t> hash_vals(rows);
// result[j] means column index, i means rows index
- for (int j = 0; j < result.size(); ++j) {
+ for (int j = 0; j < result_size; ++j) {
+ auto& column = block->get_by_position(result[j]).column;
for (int i = 0; i < rows; ++i) {
- auto val = block->get_by_position(result[j]).column->get_data_at(i);
-
+ auto val = column->get_data_at(i);
if (val.data == nullptr) {
// nullptr is treat as 0 when hash
static const int INT_VALUE = 0;
@@ -446,17 +480,12 @@ Status VDataStreamSender::send(RuntimeState* state, Block* block) {
hash_vals[i] = RawValue::zlib_crc32(&INT_VALUE, INT_TYPE, hash_vals[i]);
} else {
hash_vals[i] = RawValue::zlib_crc32(val.data, val.size,
- _partition_expr_ctxs[j]->root()->type(),
- hash_vals[i]);
+ _partition_expr_ctxs[j]->root()->type(), hash_vals[i]);
}
}
}
- for (int i = 0; i < rows; i++) {
- auto target_channel_id = hash_vals[i] % num_channels;
- RETURN_IF_ERROR(_channel_shared_ptrs[target_channel_id]->add_row(&send_block, i));
- }
-
+ RETURN_IF_ERROR(channel_add_rows(_channel_shared_ptrs, num_channels, hash_vals, rows, block));
} else {
// Range partition
// 1. caculate range
diff --git a/be/src/vec/sink/vdata_stream_sender.h b/be/src/vec/sink/vdata_stream_sender.h
index a15d39d..9d20987 100644
--- a/be/src/vec/sink/vdata_stream_sender.h
+++ b/be/src/vec/sink/vdata_stream_sender.h
@@ -72,6 +72,17 @@ public:
private:
class Channel;
+ Status get_partition_column_result(Block* block, int* result) const {
+ int counter = 0;
+ for (auto ctx : _partition_expr_ctxs) {
+ RETURN_IF_ERROR(ctx->execute(block, &result[counter++]));
+ }
+ return Status::OK();
+ }
+
+ template <typename Channels, typename HashVals>
+ Status channel_add_rows(Channels& channels, int num_channels, const HashVals& hash_vals, int rows, Block* block);
+
struct hash_128 {
uint64_t high;
uint64_t low;
@@ -179,6 +190,7 @@ public:
Status send_block(PBlock* block, bool eos = false);
Status add_row(Block* block, int row);
+ Status add_rows(Block* block, const std::vector<int>& row);
Status send_current_block(bool eos = false);
@@ -203,9 +215,9 @@ public:
return uid.to_string();
}
- TUniqueId get_fragment_instance_id() { return _fragment_instance_id; }
+ TUniqueId get_fragment_instance_id() const { return _fragment_instance_id; }
- bool is_local() { return _is_local; }
+ bool is_local() const { return _is_local; }
private:
inline Status _wait_last_brpc() {
@@ -223,6 +235,7 @@ private:
return Status::OK();
}
+
private:
// Serialize _batch into _thrift_batch and send via send_batch().
// Returns send_batch() status.
@@ -261,5 +274,24 @@ private:
size_t _capacity;
bool _is_local;
};
+
+template <typename Channels, typename HashVals>
+Status VDataStreamSender::channel_add_rows(Channels& channels, int num_channels, const HashVals& hash_vals, int rows, Block* block) {
+ std::vector<int> channel2rows[num_channels];
+
+ for (int i = 0; i < rows; i++) {
+ auto cid = hash_vals[i] % num_channels;
+ channel2rows[cid].emplace_back(i);
+ }
+
+ for (int i = 0; i < num_channels; ++i) {
+ if (!channel2rows[i].empty()) {
+ RETURN_IF_ERROR(channels[i]->add_rows(block, channel2rows[i]));
+ }
+ }
+
+ return Status::OK();
+}
+
} // namespace vectorized
} // namespace doris
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 21/33: [Vectorized][Enhancement] use simd to speed up coalesce and if_not_null function (#7722)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit f3ce1cba69aa27f8ae0af63451bab41d5048c367
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Jan 12 13:20:48 2022 +0800
[Vectorized][Enhancement] use simd to speed up coalesce and if_not_null function (#7722)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/vec/functions/function_coalesce.cpp | 210 ++++++++++++++++++++-----
be/src/vec/functions/is_not_null.cpp | 4 +-
be/src/vec/functions/simple_function_factory.h | 5 +-
be/test/vec/function/function_string_test.cpp | 42 +++++
4 files changed, 214 insertions(+), 47 deletions(-)
diff --git a/be/src/vec/functions/function_coalesce.cpp b/be/src/vec/functions/function_coalesce.cpp
index 65d544c..99b6110 100644
--- a/be/src/vec/functions/function_coalesce.cpp
+++ b/be/src/vec/functions/function_coalesce.cpp
@@ -28,6 +28,8 @@ class FunctionCoalesce : public IFunction {
public:
static constexpr auto name = "coalesce";
+ mutable FunctionBasePtr func_is_not_null;
+
static FunctionPtr create() { return std::make_shared<FunctionCoalesce>(); }
String get_name() const override { return name; }
@@ -41,47 +43,70 @@ public:
size_t get_number_of_arguments() const override { return 0; }
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ DataTypePtr res;
for (const auto& arg : arguments) {
if (!arg->is_nullable()) {
- return arg;
+ res = arg;
+ break;
}
}
- return arguments[0];
+
+ res = res ? res : arguments[0];
+
+ const ColumnsWithTypeAndName is_not_null_col{
+ {nullptr, make_nullable(res), ""}
+ };
+ func_is_not_null = SimpleFunctionFactory::instance().
+ get_function("is_not_null_pred", is_not_null_col, std::make_shared<DataTypeUInt8>());
+
+ return res;
}
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) override {
DCHECK_GE(arguments.size(), 1);
+ DataTypePtr result_type = block.get_by_position(result).type;
ColumnNumbers filtered_args;
filtered_args.reserve(arguments.size());
- for (const auto& arg : arguments) {
- const auto& type = block.get_by_position(arg).type;
- if (type->only_null()) {
- continue;
- }
- filtered_args.push_back(arg);
- if (!type->is_nullable()) {
- break;
+
+ for (size_t i = 0; i < arguments.size(); ++i) {
+ const auto& arg_type = block.get_by_position(arguments[i]).type;
+ filtered_args.push_back(arguments[i]);
+ if (!arg_type->is_nullable()) {
+ if (i == 0) { //if the first column not null, return it's directly
+ block.get_by_position(result).column = block.get_by_position(arguments[0]).column;
+ return Status::OK();
+ } else {
+ break;
+ }
}
}
size_t remaining_rows = input_rows_count;
size_t argument_size = filtered_args.size();
- std::vector<int> record_idx(input_rows_count, -1); //used to save column idx
+ std::vector<uint32_t> record_idx(input_rows_count, 0); //used to save column idx, record the result data of each row from which column
+ std::vector<uint8_t> filled_flags(input_rows_count, 0); //used to save filled flag, in order to check current row whether have filled data
+
MutableColumnPtr result_column;
+ if (!result_type->is_nullable()) {
+ result_column = result_type->create_column();
+ } else {
+ result_column = remove_nullable(result_type)->create_column();
+ }
- DataTypePtr type = block.get_by_position(result).type;
- if (!type->is_nullable()) {
- result_column = type->create_column();
+ // because now the string types does not support random position writing,
+ // so insert into result data have two methods, one is for string types, one is for others type remaining
+ bool is_string_result = result_column->is_column_string();
+ if (is_string_result) {
+ result_column->reserve(input_rows_count);
} else {
- result_column = remove_nullable(type)->create_column();
+ result_column->resize(input_rows_count);
}
- result_column->reserve(input_rows_count);
auto return_type = std::make_shared<DataTypeUInt8>();
- auto null_map = ColumnUInt8::create(input_rows_count, 1);
- auto& null_map_data = null_map->get_data();
- ColumnPtr argument_columns[argument_size];
+ auto null_map = ColumnUInt8::create(input_rows_count, 1); //if null_map_data==1, the current row should be null
+ auto* __restrict null_map_data = null_map->get_data().data();
+ ColumnPtr argument_columns[argument_size]; //use to save nested_column if is nullable column
for (size_t i = 0; i < argument_size; ++i) {
block.get_by_position(filtered_args[i]).column =
@@ -93,40 +118,69 @@ public:
}
}
+ Block temporary_block {
+ ColumnsWithTypeAndName {
+ block.get_by_position(filtered_args[0]),
+ {nullptr, std::make_shared<DataTypeUInt8>(), ""}
+ }
+ };
+
for (size_t i = 0; i < argument_size && remaining_rows; ++i) {
- const ColumnsWithTypeAndName is_not_null_col {block.get_by_position(filtered_args[i])};
- Block temporary_block(is_not_null_col);
- temporary_block.insert(ColumnWithTypeAndName {nullptr, return_type, ""});
- auto func_is_not_null = SimpleFunctionFactory::instance().get_function("is_not_null_pred", is_not_null_col, return_type);
+ temporary_block.get_by_position(0).column = block.get_by_position(filtered_args[i]).column;
func_is_not_null->execute(context, temporary_block, {0}, 1, input_rows_count);
- auto res_column = std::move(*temporary_block.get_by_position(1).column->convert_to_full_column_if_const()).mutate();
+ auto res_column = (*temporary_block.get_by_position(1).column->convert_to_full_column_if_const()).mutate();
auto& res_map = assert_cast<ColumnVector<UInt8>*>(res_column.get())->get_data();
- auto* res = res_map.data();
-
- //TODO: if need to imporve performance in the feature, here it's could SIMD
- for (size_t j = 0; j < input_rows_count && remaining_rows; ++j) {
- if (res[j] && null_map_data[j]) {
- null_map_data[j] = 0;
- remaining_rows--;
- record_idx[j] = i;
+ auto* __restrict res = res_map.data();
+
+ // Here it's SIMD thought the compiler automatically
+ // true: res[j]==1 && null_map_data[j]==1, false: others
+ // if true: remaining_rows--; record_idx[j]=column_idx; null_map_data[j]=0, so the current row could fill result
+ for (size_t j = 0; j < input_rows_count; ++j) {
+ remaining_rows -= (res[j] & null_map_data[j]);
+ record_idx[j] += (res[j] & null_map_data[j]) * i;
+ null_map_data[j] -= (res[j] & null_map_data[j]);
+ }
+
+ if (remaining_rows == 0) {
+ //check whether all result data from the same column
+ size_t is_same_column_count = 0;
+ const auto data = record_idx[0];
+ for (size_t row = 0; row < input_rows_count; ++row) {
+ is_same_column_count += (record_idx[row] == data);
+ }
+
+ if (is_same_column_count == input_rows_count) {
+ if (result_type->is_nullable()) {
+ block.get_by_position(result).column = make_nullable(argument_columns[i], false);
+ } else {
+ block.get_by_position(result).column = argument_columns[i];
+ }
+ return Status::OK();
}
}
+
+ if (!is_string_result) {
+ //if not string type, could check one column firstly,
+ //and then fill the not null value in result column,
+ //this method may result in higher CPU cache
+ filled_result_column(result_type, result_column, argument_columns[i], null_map_data,
+ filled_flags.data(), input_rows_count);
+ }
}
- //TODO: According to the record results, fill in result one by one,
- //that's fill in result use different methods for different types,
- //because now the string types does not support random position writing,
- //But other type could, so could check one column, and fill the not null value in result column,
- //and then next column, this method may result in higher CPU cache
- for (int row = 0; row < input_rows_count; ++row) {
- if (record_idx[row] == -1) {
- result_column->insert_default();
- } else {
- result_column->insert_from(*argument_columns[record_idx[row]].get(), row);
+
+ if (is_string_result) {
+ //if string type, should according to the record results, fill in result one by one,
+ for (size_t row = 0; row < input_rows_count; ++row) {
+ if (null_map_data[row]) { //should be null
+ result_column->insert_default();
+ } else {
+ result_column->insert_from(*argument_columns[record_idx[row]].get(), row);
+ }
}
}
- if (type->is_nullable()) {
+ if (result_type->is_nullable()) {
block.replace_by_position(result, ColumnNullable::create(std::move(result_column), std::move(null_map)));
} else {
block.replace_by_position(result, std::move(result_column));
@@ -134,6 +188,76 @@ public:
return Status::OK();
}
+
+ template <typename ColumnType>
+ Status insert_result_data(MutableColumnPtr& result_column, ColumnPtr& argument_column,
+ const UInt8* __restrict null_map_data, UInt8* __restrict filled_flag,
+ const size_t input_rows_count) {
+ auto* __restrict result_raw_data =
+ reinterpret_cast<ColumnType*>(result_column.get())->get_data().data();
+ auto* __restrict column_raw_data =
+ reinterpret_cast<const ColumnType*>(argument_column.get())->get_data().data();
+
+
+ // Here it's SIMD thought the compiler automatically also
+ // true: null_map_data[row]==0 && filled_idx[row]==0
+ // if true, could filled current row data into result column
+ for (size_t row = 0; row < input_rows_count; ++row) {
+ result_raw_data[row] += (!(null_map_data[row] | filled_flag[row])) * column_raw_data[row];
+ filled_flag[row] += (!(null_map_data[row] | filled_flag[row]));
+ }
+ return Status::OK();
+ }
+
+ //TODO: this function is same as case when, should be replaced by macro
+ Status filled_result_column(const DataTypePtr& data_type, MutableColumnPtr& result_column,
+ ColumnPtr& argument_column, UInt8* __restrict null_map_data,
+ UInt8* __restrict filled_flag, const size_t input_rows_count) {
+ WhichDataType which(data_type->is_nullable()
+ ? reinterpret_cast<const DataTypeNullable*>(data_type.get())
+ ->get_nested_type()
+ : data_type);
+ if (which.is_uint8()) {
+ return insert_result_data<ColumnUInt8>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_int16()) {
+ return insert_result_data<ColumnInt16>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_uint32()) {
+ return insert_result_data<ColumnUInt32>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_uint64()) {
+ return insert_result_data<ColumnUInt64>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_int8()) {
+ return insert_result_data<ColumnInt8>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_int16()) {
+ return insert_result_data<ColumnInt16>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_int32()) {
+ return insert_result_data<ColumnInt32>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_int64()) {
+ return insert_result_data<ColumnInt64>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_date_or_datetime()) {
+ return insert_result_data<ColumnVector<DateTime>>(
+ result_column, argument_column, null_map_data, filled_flag, input_rows_count);
+ } else if (which.is_float32()) {
+ return insert_result_data<ColumnFloat32>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_float64()) {
+ return insert_result_data<ColumnFloat64>(result_column, argument_column, null_map_data,
+ filled_flag, input_rows_count);
+ } else if (which.is_decimal()) {
+ return insert_result_data<ColumnDecimal<Decimal128>>(
+ result_column, argument_column, null_map_data, filled_flag, input_rows_count);
+ } else {
+ return Status::NotSupported(fmt::format("Unexpected type {} of argument of function {}",
+ data_type->get_name(), get_name()));
+ }
+ }
};
void register_function_coalesce(SimpleFunctionFactory& factory) {
diff --git a/be/src/vec/functions/is_not_null.cpp b/be/src/vec/functions/is_not_null.cpp
index 1bfeaa6..346dac1 100644
--- a/be/src/vec/functions/is_not_null.cpp
+++ b/be/src/vec/functions/is_not_null.cpp
@@ -52,8 +52,8 @@ public:
if (auto* nullable = check_and_get_column<ColumnNullable>(*elem.column)) {
/// Return the negated null map.
auto res_column = ColumnUInt8::create(input_rows_count);
- const auto& src_data = nullable->get_null_map_data();
- auto& res_data = assert_cast<ColumnUInt8&>(*res_column).get_data();
+ const auto* __restrict src_data = nullable->get_null_map_data().data();
+ auto* __restrict res_data = assert_cast<ColumnUInt8&>(*res_column).get_data().data();
for (size_t i = 0; i < input_rows_count; ++i) {
res_data[i] = !src_data[i];
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index 1d064bb..d757920 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -65,10 +65,11 @@ void register_function_like(SimpleFunctionFactory& factory);
void register_function_regexp(SimpleFunctionFactory& factory);
void register_function_random(SimpleFunctionFactory& factory);
void register_function_coalesce(SimpleFunctionFactory& factory);
+
class SimpleFunctionFactory {
using Creator = std::function<FunctionBuilderPtr()>;
- using FunctionCreators = std::unordered_map<std::string, Creator>;
- using FunctionIsVariadic = std::unordered_set<std::string>;
+ using FunctionCreators = phmap::flat_hash_map<std::string, Creator>;
+ using FunctionIsVariadic = phmap::flat_hash_set<std::string>;
public:
void register_function(const std::string& name, Creator ptr) {
diff --git a/be/test/vec/function/function_string_test.cpp b/be/test/vec/function/function_string_test.cpp
index bd2f9a7..9f1f5d7 100644
--- a/be/test/vec/function/function_string_test.cpp
+++ b/be/test/vec/function/function_string_test.cpp
@@ -681,6 +681,48 @@ TEST(function_string_test, function_unhex_test) {
vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
}
+TEST(function_string_test, function_coalesce_test) {
+ std::string func_name = "coalesce";
+ {
+ std::vector<std::any> input_types = {vectorized::TypeIndex::Int32,
+ vectorized::TypeIndex::Int32,
+ vectorized::TypeIndex::Int32};
+ DataSet data_set = {{{Null(), Null(), (int32_t)1}, {(int32_t)1}},
+ {{Null(), Null(), (int32_t)2}, {(int32_t)2}},
+ {{Null(), Null(), (int32_t)3}, {(int32_t)3}},
+ {{Null(), Null(), (int32_t)4}, {(int32_t)4}}};
+ vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
+ data_set);
+ }
+
+ {
+ std::vector<std::any> input_types = {vectorized::TypeIndex::String,
+ vectorized::TypeIndex::String,
+ vectorized::TypeIndex::Int32};
+ DataSet data_set = {
+ {{std::string("qwer"), Null(), (int32_t)1}, {std::string("qwer")}},
+ {{std::string("asdf"), Null(), (int32_t)2}, {std::string("asdf")}},
+ {{std::string("zxcv"), Null(), (int32_t)3}, {std::string("zxcv")}},
+ {{std::string("vbnm"), Null(), (int32_t)4}, {std::string("vbnm")}},
+ };
+ vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
+ data_set);
+ }
+
+ {
+ std::vector<std::any> input_types = {vectorized::TypeIndex::String,
+ vectorized::TypeIndex::String,
+ vectorized::TypeIndex::String};
+ DataSet data_set = {
+ {{Null(), std::string("abc"), std::string("hij")}, {std::string("abc")}},
+ {{Null(), std::string("def"), std::string("klm")}, {std::string("def")}},
+ {{Null(), std::string(""), std::string("xyz")}, {std::string("")}},
+ {{Null(), Null(), std::string("uvw")}, {std::string("uvw")}}};
+ vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types,
+ data_set);
+ }
+}
+
} // namespace doris
int main(int argc, char** argv) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 25/33: [Vectorized] Rebase code from master
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 20619e795832ca6947a32787f7f8937f5c6d0411
Author: lihaopeng <li...@baidu.com>
AuthorDate: Thu Jan 13 17:27:07 2022 +0800
[Vectorized] Rebase code from master
---
be/src/exec/olap_scanner.cpp | 2 +-
be/src/exec/olap_scanner.h | 4 ++--
be/src/vec/exec/join/vhash_join_node.cpp | 2 +-
be/src/vec/exec/volap_scanner.cpp | 7 ++++---
be/src/vec/exec/volap_scanner.h | 6 +++++-
be/src/vec/functions/function_binary_arithmetic.h | 2 +-
be/src/vec/olap/block_reader.cpp | 2 +-
be/src/vec/olap/block_reader.h | 4 ++--
be/src/vec/olap/vcollect_iterator.cpp | 6 +++---
be/src/vec/olap/vcollect_iterator.h | 14 +++++++-------
be/src/vec/runtime/vdatetime_value.cpp | 2 +-
be/src/vec/runtime/vdatetime_value.h | 2 +-
be/test/vec/core/block_test.cpp | 2 +-
13 files changed, 30 insertions(+), 25 deletions(-)
diff --git a/be/src/exec/olap_scanner.cpp b/be/src/exec/olap_scanner.cpp
index 34336fa..2e05c5d 100644
--- a/be/src/exec/olap_scanner.cpp
+++ b/be/src/exec/olap_scanner.cpp
@@ -176,7 +176,7 @@ Status OlapScanner::_init_tablet_reader_params(
_tablet_reader_params.rs_readers[1]->rowset()->start_version() == 2 &&
!_tablet_reader_params.rs_readers[1]->rowset()->rowset_meta()->is_segments_overlapping());
- _params.origin_return_columns = &_return_columns;
+ _tablet_reader_params.origin_return_columns = &_return_columns;
if (_aggregation || single_version) {
_tablet_reader_params.return_columns = _return_columns;
_tablet_reader_params.direct_mode = true;
diff --git a/be/src/exec/olap_scanner.h b/be/src/exec/olap_scanner.h
index f234925..0c684d9 100644
--- a/be/src/exec/olap_scanner.h
+++ b/be/src/exec/olap_scanner.h
@@ -58,7 +58,7 @@ public:
Status open();
- Status get_batch(RuntimeState* state, RowBatch* batch, bool* eof);
+ virtual Status get_batch(RuntimeState* state, RowBatch* batch, bool* eof);
Status close(RuntimeState* state);
@@ -103,7 +103,7 @@ protected:
// Update profile that need to be reported in realtime.
void _update_realtime_counter();
- virtual void set_tablet_reader() { _tablet_reader.reset(new TupleReader); }
+ virtual void set_tablet_reader() { _tablet_reader = std::make_unique<TupleReader>(); }
protected:
RuntimeState* _runtime_state;
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp b/be/src/vec/exec/join/vhash_join_node.cpp
index 4533cae..9563ebf 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -590,7 +590,7 @@ Status HashJoinNode::init(const TPlanNode& tnode, RuntimeState* state) {
for (const auto& filter_desc : _runtime_filter_descs) {
RETURN_IF_ERROR(state->runtime_filter_mgr()->regist_filter(RuntimeFilterRole::PRODUCER,
- filter_desc));
+ filter_desc, state->query_options()));
}
return Status::OK();
diff --git a/be/src/vec/exec/volap_scanner.cpp b/be/src/vec/exec/volap_scanner.cpp
index 64a51a1..1b4bb02 100644
--- a/be/src/vec/exec/volap_scanner.cpp
+++ b/be/src/vec/exec/volap_scanner.cpp
@@ -17,6 +17,8 @@
#include "vec/exec/volap_scanner.h"
+#include <memory>
+
#include "vec/columns/column_complex.h"
#include "vec/columns/column_nullable.h"
#include "vec/columns/column_string.h"
@@ -25,14 +27,13 @@
#include "vec/core/block.h"
#include "vec/exec/volap_scan_node.h"
#include "vec/exprs/vexpr_context.h"
-#include "vec/olap/block_reader.h"
#include "vec/runtime/vdatetime_value.h"
+
namespace doris::vectorized {
VOlapScanner::VOlapScanner(RuntimeState* runtime_state, VOlapScanNode* parent, bool aggregation,
bool need_agg_finalize, const TPaloScanRange& scan_range)
: OlapScanner(runtime_state, parent, aggregation, need_agg_finalize, scan_range) {
- _reader.reset(new BlockReader);
}
Status VOlapScanner::get_block(RuntimeState* state, vectorized::Block* block, bool* eof) {
@@ -50,7 +51,7 @@ Status VOlapScanner::get_block(RuntimeState* state, vectorized::Block* block, bo
do {
// Read one block from block reader
- auto res = _reader->next_block_with_aggregation(block, nullptr, nullptr, eof);
+ auto res = _tablet_reader->next_block_with_aggregation(block, nullptr, nullptr, eof);
if (res != OLAP_SUCCESS) {
std::stringstream ss;
ss << "Internal Error: read storage fail. res=" << res
diff --git a/be/src/vec/exec/volap_scanner.h b/be/src/vec/exec/volap_scanner.h
index 3c66f4d..5efaf9d 100644
--- a/be/src/vec/exec/volap_scanner.h
+++ b/be/src/vec/exec/volap_scanner.h
@@ -19,9 +19,10 @@
#include "exec/olap_scanner.h"
+#include "vec/olap/block_reader.h"
+
namespace doris {
class OlapScanNode;
-class OLAPReader;
class RuntimeProfile;
class Field;
class RowBatch;
@@ -41,6 +42,9 @@ public:
VExprContext** vconjunct_ctx_ptr() { return &_vconjunct_ctx; }
+protected:
+ virtual void set_tablet_reader() { _tablet_reader = std::make_unique<BlockReader>(); }
+
private:
// TODO: Remove this function after we finish reader vec
void _convert_row_to_block(std::vector<vectorized::MutableColumnPtr>* columns);
diff --git a/be/src/vec/functions/function_binary_arithmetic.h b/be/src/vec/functions/function_binary_arithmetic.h
index 180d2f5..f987f90 100644
--- a/be/src/vec/functions/function_binary_arithmetic.h
+++ b/be/src/vec/functions/function_binary_arithmetic.h
@@ -292,7 +292,7 @@ private:
DecimalV2Value r(b);
auto ans = Op::template apply(l, r, null_map, index);
NativeResultType result;
- memcpy(&result, &ans, sizeof(NativeResultType));
+ memcpy(&result, &ans, std::min(sizeof(result), sizeof(ans)));
return result;
}
diff --git a/be/src/vec/olap/block_reader.cpp b/be/src/vec/olap/block_reader.cpp
index ec891f9..8fda9d6 100644
--- a/be/src/vec/olap/block_reader.cpp
+++ b/be/src/vec/olap/block_reader.cpp
@@ -119,7 +119,7 @@ void BlockReader::_init_agg_state(const ReaderParams& read_params) {
}
OLAPStatus BlockReader::init(const ReaderParams& read_params) {
- Reader::init(read_params);
+ TabletReader::init(read_params);
_batch_size = read_params.runtime_state->batch_size();
auto return_column_size =
diff --git a/be/src/vec/olap/block_reader.h b/be/src/vec/olap/block_reader.h
index 0fba0be..b1bc7e8 100644
--- a/be/src/vec/olap/block_reader.h
+++ b/be/src/vec/olap/block_reader.h
@@ -32,9 +32,9 @@ namespace doris {
namespace vectorized {
-class BlockReader final : public Reader {
+class BlockReader final : public TabletReader {
public:
- ~BlockReader();
+ ~BlockReader() override;
// Initialize BlockReader with tablet, data version and fetch range.
OLAPStatus init(const ReaderParams& read_params) override;
diff --git a/be/src/vec/olap/vcollect_iterator.cpp b/be/src/vec/olap/vcollect_iterator.cpp
index 2d132a6..73ef823 100644
--- a/be/src/vec/olap/vcollect_iterator.cpp
+++ b/be/src/vec/olap/vcollect_iterator.cpp
@@ -24,7 +24,7 @@ namespace vectorized {
VCollectIterator::~VCollectIterator() {}
-void VCollectIterator::init(Reader* reader) {
+void VCollectIterator::init(TabletReader* reader) {
_reader = reader;
// when aggregate is enabled or key_type is DUP_KEYS, we don't merge
// multiple data to aggregate for better performance
@@ -159,7 +159,7 @@ OLAPStatus VCollectIterator::next(Block* block) {
}
}
-VCollectIterator::Level0Iterator::Level0Iterator(RowsetReaderSharedPtr rs_reader, Reader* reader)
+VCollectIterator::Level0Iterator::Level0Iterator(RowsetReaderSharedPtr rs_reader, TabletReader* reader)
: LevelIterator(reader), _rs_reader(rs_reader), _reader(reader) {
DCHECK_EQ(RowsetReader::BETA, rs_reader->type());
_block = _schema.create_block(_reader->_return_columns);
@@ -207,7 +207,7 @@ OLAPStatus VCollectIterator::Level0Iterator::next(Block* block) {
}
VCollectIterator::Level1Iterator::Level1Iterator(
- const std::list<VCollectIterator::LevelIterator*>& children, Reader* reader, bool merge,
+ const std::list<VCollectIterator::LevelIterator*>& children, TabletReader* reader, bool merge,
bool skip_same)
: LevelIterator(reader),
_children(children),
diff --git a/be/src/vec/olap/vcollect_iterator.h b/be/src/vec/olap/vcollect_iterator.h
index c517de2..6cf9d62 100644
--- a/be/src/vec/olap/vcollect_iterator.h
+++ b/be/src/vec/olap/vcollect_iterator.h
@@ -41,7 +41,7 @@ public:
// Hold reader point to get reader params
~VCollectIterator();
- void init(Reader* reader);
+ void init(TabletReader* reader);
OLAPStatus add_child(RowsetReaderSharedPtr rs_reader);
@@ -69,7 +69,7 @@ private:
// then merged with other rowset readers.
class LevelIterator {
public:
- LevelIterator(Reader* reader) : _schema(reader->tablet()->tablet_schema()) {};
+ LevelIterator(TabletReader* reader) : _schema(reader->tablet()->tablet_schema()) {};
virtual OLAPStatus init() = 0;
@@ -112,7 +112,7 @@ private:
// Iterate from rowset reader. This Iterator usually like a leaf node
class Level0Iterator : public LevelIterator {
public:
- Level0Iterator(RowsetReaderSharedPtr rs_reader, Reader* reader);
+ Level0Iterator(RowsetReaderSharedPtr rs_reader, TabletReader* reader);
~Level0Iterator() {}
OLAPStatus init() override;
@@ -127,14 +127,14 @@ private:
OLAPStatus _refresh_current_row();
RowsetReaderSharedPtr _rs_reader;
- Reader* _reader = nullptr;
+ TabletReader* _reader = nullptr;
Block _block;
};
// Iterate from LevelIterators (maybe Level0Iterators or Level1Iterator or mixed)
class Level1Iterator : public LevelIterator {
public:
- Level1Iterator(const std::list<LevelIterator*>& children, Reader* reader, bool merge,
+ Level1Iterator(const std::list<LevelIterator*>& children, TabletReader* reader, bool merge,
bool skip_same);
OLAPStatus init() override;
@@ -160,7 +160,7 @@ private:
// point to the Level0Iterator containing the next output row.
// null when VCollectIterator hasn't been initialized or reaches EOF.
LevelIterator* _cur_child = nullptr;
- Reader* _reader = nullptr;
+ TabletReader* _reader = nullptr;
// when `_merge == true`, rowset reader returns ordered rows and VCollectIterator uses a priority queue to merge
// sort them. The output of VCollectIterator is also ordered.
@@ -184,7 +184,7 @@ private:
bool _merge = true;
// Hold reader point to access read params, such as fetch conditions.
- Reader* _reader = nullptr;
+ TabletReader* _reader = nullptr;
bool _skip_same;
};
diff --git a/be/src/vec/runtime/vdatetime_value.cpp b/be/src/vec/runtime/vdatetime_value.cpp
index cc4cc0e..4c59277 100644
--- a/be/src/vec/runtime/vdatetime_value.cpp
+++ b/be/src/vec/runtime/vdatetime_value.cpp
@@ -292,7 +292,7 @@ void VecDateTimeValue::set_type(int type) {
void VecDateTimeValue::set_max_time(bool neg) {
set_zero(TIME_TIME);
- _hour = TIME_MAX_HOUR;
+ _hour = static_cast<uint8_t>(TIME_MAX_HOUR);
_minute = TIME_MAX_MINUTE;
_second = TIME_MAX_SECOND;
_neg = neg;
diff --git a/be/src/vec/runtime/vdatetime_value.h b/be/src/vec/runtime/vdatetime_value.h
index a008f0c..4ec8d6e 100644
--- a/be/src/vec/runtime/vdatetime_value.h
+++ b/be/src/vec/runtime/vdatetime_value.h
@@ -127,7 +127,7 @@ const int DATE_MAX_DAYNR = 3652424;
const int YY_PART_YEAR = 70;
// Limits of time value
-const int TIME_MAX_HOUR = 838;
+const int TIME_MAX_HOUR = 256;
const int TIME_MAX_MINUTE = 59;
const int TIME_MAX_SECOND = 59;
const int TIME_MAX_VALUE = 10000 * TIME_MAX_HOUR + 100 * TIME_MAX_MINUTE + TIME_MAX_SECOND;
diff --git a/be/test/vec/core/block_test.cpp b/be/test/vec/core/block_test.cpp
index 711f29f..7264369 100644
--- a/be/test/vec/core/block_test.cpp
+++ b/be/test/vec/core/block_test.cpp
@@ -140,7 +140,7 @@ TEST(BlockTest, RowBatchCovertToBlock) {
larget_int = column7->operator[](i).get<vectorized::Int128>();
vectorized::VecDateTimeValue k7;
- memcpy(&k7, &larget_int, column_descs[6].size);
+ memcpy(reinterpret_cast<vectorized::Int128*>(&k7), &larget_int, column_descs[6].size);
vectorized::VecDateTimeValue date_time_value;
std::string now_time("2020-12-02");
date_time_value.from_date_str(now_time.c_str(), now_time.size());
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 19/33: [Vectorized][Bug] fix 'negative' function ut run fail && fix testIsBucketShuffleJoin run fail && fix some compile fail (#7688)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 9f16ac2363f2777d55e7538eda2daee46df643b0
Author: Pxl <95...@qq.com>
AuthorDate: Tue Jan 11 10:48:02 2022 +0800
[Vectorized][Bug] fix 'negative' function ut run fail && fix testIsBucketShuffleJoin run fail && fix some compile fail (#7688)
---
be/src/util/brpc_stub_cache.h | 2 +-
be/test/vec/function/function_hash_test.cpp | 36 +++++++++++++---------
be/test/vec/function/function_math_test.cpp | 33 ++++++++------------
.../org/apache/doris/planner/HashJoinNode.java | 2 +-
.../java/org/apache/doris/qe/CoordinatorTest.java | 9 ++++--
5 files changed, 44 insertions(+), 38 deletions(-)
diff --git a/be/src/util/brpc_stub_cache.h b/be/src/util/brpc_stub_cache.h
index e944aac..21800f3 100644
--- a/be/src/util/brpc_stub_cache.h
+++ b/be/src/util/brpc_stub_cache.h
@@ -47,7 +47,7 @@ namespace doris {
class BrpcStubCache {
public:
BrpcStubCache();
- ~BrpcStubCache();
+ virtual ~BrpcStubCache();
inline std::shared_ptr<PBackendService_Stub> get_stub(const butil::EndPoint& endpoint) {
auto stub_ptr = _stub_map.find(endpoint);
diff --git a/be/test/vec/function/function_hash_test.cpp b/be/test/vec/function/function_hash_test.cpp
index 75a2ac5..d5d41b2 100644
--- a/be/test/vec/function/function_hash_test.cpp
+++ b/be/test/vec/function/function_hash_test.cpp
@@ -33,20 +33,21 @@ TEST(HashFunctionTest, murmur_hash_3_test) {
{
std::vector<std::any> input_types = {vectorized::TypeIndex::String};
- DataSet data_set = {{{Null()}, Null()},
- {{std::string("hello")}, (int32_t) 1321743225}};
+ DataSet data_set = {{{Null()}, Null()}, {{std::string("hello")}, (int32_t)1321743225}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
+ data_set);
};
{
std::vector<std::any> input_types = {vectorized::TypeIndex::String,
vectorized::TypeIndex::String};
- DataSet data_set = {{{std::string("hello"), std::string("world")}, (int32_t) 984713481},
+ DataSet data_set = {{{std::string("hello"), std::string("world")}, (int32_t)984713481},
{{std::string("hello"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
+ data_set);
};
{
@@ -54,10 +55,12 @@ TEST(HashFunctionTest, murmur_hash_3_test) {
vectorized::TypeIndex::String,
vectorized::TypeIndex::String};
- DataSet data_set = {{{std::string("hello"), std::string("world"), std::string("!")}, (int32_t) -666935433},
+ DataSet data_set = {{{std::string("hello"), std::string("world"), std::string("!")},
+ (int32_t)-666935433},
{{std::string("hello"), std::string("world"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types, data_set);
+ vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
+ data_set);
};
}
@@ -68,19 +71,22 @@ TEST(HashFunctionTest, murmur_hash_2_test) {
std::vector<std::any> input_types = {vectorized::TypeIndex::String};
DataSet data_set = {{{Null()}, Null()},
- {{std::string("hello")}, (uint64_t) 2191231550387646743}};
+ {{std::string("hello")}, (uint64_t)2191231550387646743ull}};
- vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types, data_set);
+ vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types,
+ data_set);
};
{
std::vector<std::any> input_types = {vectorized::TypeIndex::String,
vectorized::TypeIndex::String};
- DataSet data_set = {{{std::string("hello"), std::string("world")}, (uint64_t) 11978658642541747642l},
- {{std::string("hello"), Null()}, Null()}};
+ DataSet data_set = {
+ {{std::string("hello"), std::string("world")}, (uint64_t)11978658642541747642ull},
+ {{std::string("hello"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types, data_set);
+ vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types,
+ data_set);
};
{
@@ -88,10 +94,12 @@ TEST(HashFunctionTest, murmur_hash_2_test) {
vectorized::TypeIndex::String,
vectorized::TypeIndex::String};
- DataSet data_set = {{{std::string("hello"), std::string("world"), std::string("!")}, (uint64_t) 1367324781703025231},
+ DataSet data_set = {{{std::string("hello"), std::string("world"), std::string("!")},
+ (uint64_t)1367324781703025231ull},
{{std::string("hello"), std::string("world"), Null()}, Null()}};
- vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types, data_set);
+ vectorized::check_function<vectorized::DataTypeUInt64, true>(func_name, input_types,
+ data_set);
};
}
diff --git a/be/test/vec/function/function_math_test.cpp b/be/test/vec/function/function_math_test.cpp
index 0413abd..17748a0 100644
--- a/be/test/vec/function/function_math_test.cpp
+++ b/be/test/vec/function/function_math_test.cpp
@@ -306,9 +306,9 @@ TEST(MathFunctionTest, negative_test) {
std::vector<std::any> input_types = {vectorized::TypeIndex::Int32};
DataSet data_set = {{{(int32_t)3}, (int32_t)-3},
- {{(int32_t)-3}, (int32_t)-3},
+ {{(int32_t)-3}, (int32_t)3},
{{(int32_t)0}, (int32_t)0},
- {{(int32_t)-60}, (int32_t)-60}};
+ {{(int32_t)-60}, (int32_t)60}};
vectorized::check_function<vectorized::DataTypeInt32, true>(func_name, input_types,
data_set);
@@ -344,12 +344,8 @@ TEST(MathFunctionTest, round_test) {
{
std::vector<std::any> input_types = {vectorized::TypeIndex::Float64};
- DataSet data_set = {{{30.1}, (int64_t)30},
- {{90.6}, (int64_t)91},
- {{Null()}, Null()},
- {{0.0}, (int64_t)0},
- {{-1.1}, (int64_t)-1},
- {{-60.7}, (int64_t)-61}};
+ DataSet data_set = {{{30.1}, (int64_t)30}, {{90.6}, (int64_t)91}, {{Null()}, Null()},
+ {{0.0}, (int64_t)0}, {{-1.1}, (int64_t)-1}, {{-60.7}, (int64_t)-61}};
vectorized::check_function<vectorized::DataTypeInt64, true>(func_name, input_types,
data_set);
@@ -358,12 +354,8 @@ TEST(MathFunctionTest, round_test) {
std::vector<std::any> input_types = {vectorized::TypeIndex::Float64,
vectorized::TypeIndex::Int32};
- DataSet data_set = {{{3.1415926, 2}, 3.14},
- {{3.1415926, 3}, 3.142},
- {{Null(), -2}, Null()},
- {{193.0, -2}, 200.0},
- {{193.0, -1}, 190.0},
- {{193.0, -3}, 0.0}};
+ DataSet data_set = {{{3.1415926, 2}, 3.14}, {{3.1415926, 3}, 3.142}, {{Null(), -2}, Null()},
+ {{193.0, -2}, 200.0}, {{193.0, -1}, 190.0}, {{193.0, -3}, 0.0}};
vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
data_set);
@@ -375,9 +367,9 @@ TEST(MathFunctionTest, bin_test) {
std::vector<std::any> input_types = {vectorized::TypeIndex::Int64};
- DataSet data_set = {{{(int64_t) 10}, std::string("1010")},
- {{(int64_t) 1}, std::string("1")},
- {{(int64_t) 0}, std::string("0")},
+ DataSet data_set = {{{(int64_t)10}, std::string("1010")},
+ {{(int64_t)1}, std::string("1")},
+ {{(int64_t)0}, std::string("0")},
{{Null()}, Null()}};
vectorized::check_function<vectorized::DataTypeString, true>(func_name, input_types, data_set);
@@ -385,7 +377,7 @@ TEST(MathFunctionTest, bin_test) {
TEST(MathFunctionTest, hex_test) {
std::string func_name = "hex"; // hex(int)
-
+
std::vector<std::any> input_types = {vectorized::TypeIndex::Int64};
DataSet data_set = {{{Null()}, Null()},
@@ -413,8 +405,9 @@ TEST(MathFunctionTest, random_test) {
for (const auto& data : data_set) {
DataSet data_line = {data};
- vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types, data_line);
- }
+ vectorized::check_function<vectorized::DataTypeFloat64, true>(func_name, input_types,
+ data_line);
+ }
}
} // namespace doris
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
index aa3662f..2ee2618 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/HashJoinNode.java
@@ -91,7 +91,7 @@ public class HashJoinNode extends PlanNode {
if (joinOp.equals(JoinOperator.LEFT_ANTI_JOIN) || joinOp.equals(JoinOperator.LEFT_SEMI_JOIN)
|| joinOp.equals(JoinOperator.NULL_AWARE_LEFT_ANTI_JOIN)) {
tupleIds.addAll(outer.getTupleIds());
- } else if (joinOp.equals(JoinOperator.RIGHT_ANTI_JOIN) || joinOp.equals(JoinOperator.RIGHT_SEMI_JOIN)){
+ } else if (joinOp.equals(JoinOperator.RIGHT_ANTI_JOIN) || joinOp.equals(JoinOperator.RIGHT_SEMI_JOIN)) {
tupleIds.addAll(inner.getTupleIds());
} else {
tupleIds.addAll(outer.getTupleIds());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/CoordinatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/CoordinatorTest.java
index 3f7e923..fb86df6 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/CoordinatorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/CoordinatorTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.qe;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.BinaryPredicate;
+import org.apache.doris.analysis.BoolLiteral;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.TableRef;
import org.apache.doris.analysis.TupleDescriptor;
@@ -162,11 +163,15 @@ public class CoordinatorTest extends Coordinator {
tupleIdArrayList.add(testTupleId);
ArrayList<Expr> testJoinexprs = new ArrayList<>();
- BinaryPredicate binaryPredicate = new BinaryPredicate();
+
+ BinaryPredicate binaryPredicate = new BinaryPredicate(BinaryPredicate.Operator.EQ, new BoolLiteral(true),
+ new BoolLiteral(true));
testJoinexprs.add(binaryPredicate);
HashJoinNode hashJoinNode = new HashJoinNode(testPlanNodeId, new EmptySetNode(testPlanNodeId, tupleIdArrayList),
- new EmptySetNode(testPlanNodeId, tupleIdArrayList), new TableRef(), testJoinexprs, new ArrayList<>());
+ new EmptySetNode(testPlanNodeId, tupleIdArrayList), new TableRef(), testJoinexprs,
+ new ArrayList<>());
+
hashJoinNode.setFragment(new PlanFragment(new PlanFragmentId(-1), hashJoinNode,
new DataPartition(TPartitionType.BUCKET_SHFFULE_HASH_PARTITIONED, testJoinexprs)));
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 32/33: [Vectorized][Bug] Fix bug of repeated node resize and compile failed (#7778)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit b634ea62bbdef860a5bafffb078e6f128bfaaaff
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Mon Jan 17 20:11:26 2022 +0800
[Vectorized][Bug] Fix bug of repeated node resize and compile failed (#7778)
Co-authored-by: lihaopeng <li...@baidu.com>
---
be/src/vec/exec/vrepeat_node.cpp | 13 ++++++-------
be/src/vec/functions/simple_function_factory.h | 2 +-
build.sh | 2 +-
3 files changed, 8 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/exec/vrepeat_node.cpp b/be/src/vec/exec/vrepeat_node.cpp
index dd8bb28..287aa6e 100644
--- a/be/src/vec/exec/vrepeat_node.cpp
+++ b/be/src/vec/exec/vrepeat_node.cpp
@@ -104,6 +104,7 @@ Status VRepeatNode::get_repeated_block(Block* child_block, int repeat_id_idx, Bl
std::set<SlotId>& repeat_ids = _slot_id_set_list[repeat_id_idx];
bool is_repeat_slot = _all_slot_ids.find(_output_slots[cur_col]->id()) != _all_slot_ids.end();
bool is_set_null_slot = repeat_ids.find(_output_slots[cur_col]->id()) == repeat_ids.end();
+ const auto column_size = src_column.column->size();
if (is_repeat_slot) {
DCHECK(_output_slots[cur_col]->is_nullable());
@@ -113,21 +114,19 @@ Status VRepeatNode::get_repeated_block(Block* child_block, int repeat_id_idx, Bl
// set slot null not in repeat_ids
if (is_set_null_slot) {
- nullable_column->resize(src_column.column->size());
- for (size_t j = 0; j < src_column.column->size(); ++j) {
- nullable_column->insert_data(nullptr, 0);
- }
+ nullable_column->resize(column_size);
+ memset(nullable_column->get_null_map_data().data(), 1, sizeof(UInt8) * column_size);
} else {
if (!src_column.type->is_nullable()) {
- for (size_t j = 0; j < src_column.column->size(); ++j) {
+ for (size_t j = 0; j < column_size; ++j) {
null_map.push_back(0);
}
column_ptr = &nullable_column->get_nested_column();
}
- column_ptr->insert_range_from(*src_column.column, 0, src_column.column->size());
+ column_ptr->insert_range_from(*src_column.column, 0, column_size);
}
} else {
- columns[cur_col]->insert_range_from(*src_column.column, 0, src_column.column->size());
+ columns[cur_col]->insert_range_from(*src_column.column, 0, column_size);
}
cur_col++;
}
diff --git a/be/src/vec/functions/simple_function_factory.h b/be/src/vec/functions/simple_function_factory.h
index 45420cb..5b9f0fd 100644
--- a/be/src/vec/functions/simple_function_factory.h
+++ b/be/src/vec/functions/simple_function_factory.h
@@ -66,7 +66,7 @@ void register_function_like(SimpleFunctionFactory& factory);
void register_function_regexp(SimpleFunctionFactory& factory);
void register_function_random(SimpleFunctionFactory& factory);
void register_function_coalesce(SimpleFunctionFactory& factory);
-+void register_function_grouping(SimpleFunctionFactory& factory);
+void register_function_grouping(SimpleFunctionFactory& factory);
class SimpleFunctionFactory {
using Creator = std::function<FunctionBuilderPtr()>;
diff --git a/build.sh b/build.sh
index d842b07..a5ccaae 100755
--- a/build.sh
+++ b/build.sh
@@ -104,7 +104,7 @@ fi
eval set -- "$OPTS"
-PARALLEL=$[$(nproc)+1]
+PARALLEL=$[$(nproc)/4+1]
BUILD_BE=
BUILD_FE=
BUILD_UI=
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 03/33: [Function] Fix error about rank/dense_rank/row_number return always not nullable (#7561)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch vectorized
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 9431a93dd623f76618c31921ea1f650ce6444a6a
Author: zhangstar333 <87...@users.noreply.github.com>
AuthorDate: Tue Jan 4 11:05:32 2022 +0800
[Function] Fix error about rank/dense_rank/row_number return always not nullable (#7561)
---
.../src/main/java/org/apache/doris/catalog/AggregateFunction.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/AggregateFunction.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/AggregateFunction.java
index 4e85ca3..82e4035 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/AggregateFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/AggregateFunction.java
@@ -49,7 +49,7 @@ public class AggregateFunction extends Function {
private static final Logger LOG = LogManager.getLogger(AggregateFunction.class);
public static ImmutableSet<String> NOT_NULLABLE_AGGREGATE_FUNCTION_NAME_SET =
- ImmutableSet.of(FunctionSet.COUNT, "ndv", FunctionSet.BITMAP_UNION_INT, FunctionSet.BITMAP_UNION_COUNT, "ndv_no_finalize");
+ ImmutableSet.of("row_number", "rank", "dense_rank", FunctionSet.COUNT, "ndv", FunctionSet.BITMAP_UNION_INT, FunctionSet.BITMAP_UNION_COUNT, "ndv_no_finalize");
// Set if different from retType_, null otherwise.
private Type intermediateType;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org