You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/11/09 14:55:29 UTC
[doris] branch branch-1.2-unstable updated (c3172a559b -> 3fff8f399b)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
discard c3172a559b [fix](build) fix compile fail on Segment::open (#14058)
discard 836021ac2e [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null (#13978)
discard 3da8dfb658 [feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF (#13994)
discard 7e9890ea6d [fix](Nereids) Use simple cost to calculate benefit and avoid unuseless calculation (#14056)
discard b4857cb608 [feature-wip](multi-catalog) fix page index filter bug (#14015)
discard 944d572053 [thirdpart](lib) Add lock free queue of concurrentqueue (#14045)
discard df2f528781 [feature](Nereids) support query that group by use alias generated in aggregate output (#14030)
discard 73a004c8dc [Enhancement](Dictionary-codec) update dict once on same segment (#13936)
discard bcde11d82d [fix](priv) fix meta replay bug when upgrading from 1.1.x to 1.2.x (#14046)
new ab9aef389f [fix](priv) fix meta replay bug when upgrading from 1.1.x to 1.2.x (#14046)
new f4b4e60349 [Enhancement](Dictionary-codec) update dict once on same segment (#13936)
new 5db650f961 [feature](Nereids) support query that group by use alias generated in aggregate output (#14030)
new 89951f3f91 [thirdpart](lib) Add lock free queue of concurrentqueue (#14045)
new 2d8d8233b6 [feature-wip](multi-catalog) fix page index filter bug (#14015)
new 1979312b47 [fix](Nereids) Use simple cost to calculate benefit and avoid unuseless calculation (#14056)
new 2e0af89b7e [feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF (#13994)
new 682dc378dd [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null (#13978)
new 8aa048c056 [fix](build) fix compile fail on Segment::open (#14058)
new dd2a122707 [regression](Nereids) add back tpch regression test cases (#13826)
new 6d699fbcb5 [enhancement](Nereids) tpch q21 anti and semi join reorder (#14037)
new 2367e098d2 [chore](bin) do not set heap limit for tcmalloc until doris does not allocates large unused memory (#13761)
new bbc60f4e99 [fix](statistics) ColumnStatistics was changed unexpectedly when show stats (#14068)
new 76b3c2b4a4 [improvement](profile) support ordinary user to get query profile via http api (#14016)
new 0e39243080 [Nereids][Improve] infer predicate after push down predicate (#12996)
new 087f168f2f [fix](keyranges) fix the split error of keyranges (#14049)
new f56fe5ff35 use extern template to date_time_add (#13970)
new 81d62d6c93 [feature](information_schema) add `backends` information_schema table (#13086)
new c84ab79705 [feature](inverted index)WIP inverted index api: SQL syntax and metadata (#13430)
new bd2daec2ef [opt](ssb) Add query hint for the SSB queries (#14089)
new f4d2c9aaab [refactor](new-scan) remove old vectorized scan node (#14029)
new 44df5b384a [docs](odbc) fix docs for sqlserver odbc table (#14017)
new 83f243acf0 [enhancement](load) shrink reserved buffer for page builder (#14012) (#14014)
new 6104991317 [typo](docs)update array type doc #14057
new 83663081fe [fix](JSON) Fail to parse JSONPath (libc++) (#13941)
new ddb57a4eeb [fix](ctas) text column type len = 1 when create table as select (#13906)
new b5db5c1f0a [bug](ColumnDecimal)call set_decimalv2_type when cloning ColumnDecimal (#14061)
new a02be24c40 [fix](Vectorized)fix json_object and json_array function return wrong result on vectorized engine (#13775)
new f5ec89b45b [Compile](join) Boost compiling and linking (#14081)
new fdb42a4dd2 [docs](array-type) update the docs to specify how to use array function when import data (#13995)
new 0ef2377d22 [feature](Nereids) binding slot in order by that not show in project (#14042)
new e91dae0198 [improve](Nereids): remove redundant code, add annotation in Memo. (#14083)
new dd2e4b0425 [fix](Nereids) aggregate disassemble generate error output list on GLOBAL phase aggregate (#14079)
new 1eabf6975a [performance-wip] (vectorization) Opt HashJoin Performance (#12390)
new 12ca106cae [fix](compile) fix compile error #14103
new 09650f22e5 [feature](table-valued-function) Support `desc from s3()` and modify the syntax of tvf (#14047)
new e8a9151f13 [enhancement](Nereids) use post-order to generate runtime filter in RuntimeFilterGenerator (#13949)
new 6f022b40a2 [refractor](array) refractor DataTypeArray from_string (#13905)
new 3fff8f399b [Enhancement][fix](profile)() modify some profiles (#14074)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (c3172a559b)
\
N -- N -- N refs/heads/branch-1.2-unstable (3fff8f399b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 39 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/common/config.h | 4 -
be/src/exec/CMakeLists.txt | 1 +
be/src/exec/broker_scanner.cpp | 4 -
be/src/exec/es_http_scan_node.cpp | 15 +-
be/src/exec/es_http_scan_node.h | 6 -
be/src/exec/exec_node.cpp | 37 +-
be/src/exec/olap_common.h | 28 +-
be/src/exec/olap_scan_node.h | 1 -
be/src/exec/schema_scan_node.cpp | 5 +
be/src/exec/schema_scanner.cpp | 47 +-
be/src/exec/schema_scanner.h | 10 +
.../schema_scanner/schema_backends_scanner.cpp | 165 ++
.../schema_scanner/schema_backends_scanner.h} | 36 +-
be/src/exprs/expr_context.h | 5 -
be/src/exprs/json_functions.cpp | 12 +-
be/src/exprs/json_functions.h | 5 +-
be/src/olap/CMakeLists.txt | 1 +
be/src/olap/inverted_index_parser.cpp | 65 +
.../options.h => inverted_index_parser.h} | 30 +-
be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 8 +-
be/src/olap/rowset/segment_v2/binary_plain_page.h | 14 +-
be/src/olap/rowset/segment_v2/options.h | 2 +
be/src/olap/tablet_meta.cpp | 35 +
be/src/olap/tablet_schema.cpp | 120 ++
be/src/olap/tablet_schema.h | 29 +
be/src/util/string_util.h | 13 +-
be/src/vec/CMakeLists.txt | 6 -
be/src/vec/columns/column_decimal.cpp | 3 +
be/src/vec/common/hash_table/join_hash_map.h | 133 ++
be/src/vec/common/hash_table/join_hash_table.h | 733 ++++++++
be/src/vec/data_types/data_type_array.cpp | 159 +-
be/src/vec/exec/join/vhash_join_node.cpp | 264 +--
be/src/vec/exec/join/vhash_join_node.h | 60 +-
be/src/vec/exec/scan/new_olap_scan_node.cpp | 9 +
be/src/vec/exec/ves_http_scan_node.cpp | 447 -----
be/src/vec/exec/ves_http_scan_node.h | 113 --
be/src/vec/exec/ves_http_scanner.cpp | 52 -
be/src/vec/exec/vjdbc_connector.cpp | 2 +-
be/src/vec/exec/vjdbc_connector.h | 2 +-
be/src/vec/exec/vjdbc_scan_node.cpp | 173 --
be/src/vec/exec/vjdbc_scan_node.h | 68 -
be/src/vec/exec/vmysql_scan_node.h | 1 +
be/src/vec/exec/vodbc_scan_node.cpp | 249 ---
be/src/vec/exec/vodbc_scan_node.h | 87 -
be/src/vec/exec/volap_scan_node.cpp | 1843 --------------------
be/src/vec/exec/volap_scan_node.h | 346 ----
be/src/vec/exec/volap_scanner.cpp | 477 -----
be/src/vec/exec/volap_scanner.h | 153 --
be/src/vec/exec/vschema_scan_node.cpp | 24 +-
be/src/vec/exprs/vliteral.cpp | 2 +
.../function_date_or_datetime_computation.h | 2 +-
be/src/vec/functions/function_json.cpp | 100 +-
be/test/CMakeLists.txt | 1 +
be/test/exec/olap_common_test.cpp | 2 +-
be/test/vec/columns/column_decimal_test.cpp | 53 +
bin/start_be.sh | 2 +-
dist/LICENSE-dist.txt | 1 +
.../fe/manager/query-profile-action.md | 101 ++
docs/en/docs/data-operate/import/load-manual.md | 17 +
.../docs/ecosystem/external-table/odbc-of-doris.md | 26 +-
.../sql-functions/table-functions/numbers.md | 11 +-
.../sql-manual/sql-reference/Data-Types/ARRAY.md | 12 +-
.../fe/manager/query-profile-action.md | 102 ++
docs/zh-CN/docs/data-operate/import/load-manual.md | 18 +
.../docs/ecosystem/external-table/odbc-of-doris.md | 27 +-
.../sql-functions/table-functions/numbers.md | 11 +-
.../sql-manual/sql-reference/Data-Types/ARRAY.md | 13 +-
fe/fe-core/src/main/cup/sql_parser.cup | 23 +-
.../java/org/apache/doris/analysis/ColumnDef.java | 7 +-
.../apache/doris/analysis/CreateIndexClause.java | 6 +-
.../org/apache/doris/analysis/CreateTableStmt.java | 5 +-
.../org/apache/doris/analysis/DescribeStmt.java | 46 +-
.../java/org/apache/doris/analysis/IndexDef.java | 52 +-
.../apache/doris/analysis/InvertedIndexUtil.java | 56 +
.../org/apache/doris/analysis/SchemaTableType.java | 3 +-
.../org/apache/doris/analysis/ShowIndexStmt.java | 1 +
.../doris/analysis/TableValuedFunctionRef.java | 4 +-
.../main/java/org/apache/doris/catalog/Index.java | 56 +-
.../java/org/apache/doris/catalog/SchemaTable.java | 51 +-
.../main/java/org/apache/doris/common/Config.java | 4 +-
.../apache/doris/common/util/ProfileManager.java | 25 +-
.../apache/doris/datasource/InternalCatalog.java | 3 +-
.../httpv2/rest/manager/QueryProfileAction.java | 63 +-
.../glue/translator/RuntimeFilterTranslator.java | 5 +-
.../org/apache/doris/nereids/jobs/JobType.java | 1 +
.../doris/nereids/jobs/batch/AnalyzeRulesJob.java | 4 +-
.../doris/nereids/jobs/batch/BatchRulesJob.java | 7 +
.../jobs/batch/NereidsRewriteJobExecutor.java | 3 +
.../nereids/jobs/rewrite/VisitorRewriteJob.java | 56 +
.../java/org/apache/doris/nereids/memo/Group.java | 69 +-
.../apache/doris/nereids/memo/GroupExpression.java | 25 +-
.../java/org/apache/doris/nereids/memo/Memo.java | 66 +-
.../nereids/processor/post/PlanPostProcessors.java | 2 +-
.../processor/post/RuntimeFilterContext.java | 62 +-
.../processor/post/RuntimeFilterGenerator.java | 130 +-
.../org/apache/doris/nereids/rules/RuleType.java | 7 +-
.../nereids/rules/analysis/BindSlotReference.java | 11 +-
.../nereids/rules/analysis/CheckAnalysis.java | 21 +-
...egateFunctions.java => FillUpMissingSlots.java} | 31 +-
.../rules/rewrite/AggregateDisassemble.java | 9 +-
.../rules/rewrite/logical/InferPredicates.java | 117 ++
.../rewrite/logical/PredicatePropagation.java | 105 ++
.../rules/rewrite/logical/PullUpPredicates.java | 165 ++
.../apache/doris/nereids/stats/JoinEstimation.java | 21 +-
.../nereids/trees/expressions/Expression.java | 4 +
.../trees/expressions/functions/agg/Avg.java | 2 +-
.../trees/expressions/functions/agg/Sum.java | 4 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 6 +
.../trees/plans/physical/RuntimeFilter.java | 44 -
.../apache/doris/nereids/types/DecimalType.java | 4 +-
.../java/org/apache/doris/nereids/util/Utils.java | 8 +
.../org/apache/doris/planner/SchemaScanNode.java | 2 +
.../java/org/apache/doris/qe/ShowExecutor.java | 2 +-
.../apache/doris/service/FrontendServiceImpl.java | 126 ++
.../apache/doris/statistics/OlapTableStats.java | 46 -
.../org/apache/doris/statistics/TableStats.java | 2 +-
.../org/apache/doris/statistics/TabletStats.java | 42 -
.../tablefunction/NumbersTableValuedFunction.java | 45 +-
.../doris/tablefunction/S3TableValuedFunction.java | 39 +-
.../doris/tablefunction/TableValuedFunctionIf.java | 4 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
.../doris/analysis/CreateIndexClauseTest.java | 2 +-
.../org/apache/doris/analysis/IndexDefTest.java | 6 +-
.../org/apache/doris/catalog/OlapTableTest.java | 4 +-
.../datasets/clickbench/AnalyzeClickBenchTest.java | 16 +-
.../nereids/datasets/tpch/AnalyzeTPCHTest.java | 5 +-
.../doris/nereids/datasets/tpch/TPCHUtils.java | 32 +-
.../org/apache/doris/nereids/memo/MemoTest.java | 8 +-
...ctionsTest.java => FillUpMissingSlotsTest.java} | 2 +-
.../nereids/postprocess/RuntimeFilterTest.java | 7 +-
.../rewrite/logical/AggregateDisassembleTest.java | 63 +
.../rules/rewrite/logical/InferPredicatesTest.java | 531 ++++++
.../nereids/trees/expressions/GetDataTypeTest.java | 2 +-
.../doris/nereids/util/LogicalPlanBuilder.java | 17 +-
.../persist/TableAddOrDropColumnsInfoTest.java | 2 +-
gensrc/proto/olap_file.proto | 15 +
gensrc/script/doris_builtins_functions.py | 4 +-
gensrc/thrift/Data.thrift | 9 +-
gensrc/thrift/Descriptors.thrift | 9 +-
gensrc/thrift/FrontendService.thrift | 16 +
gensrc/thrift/PlanNodes.thrift | 8 +
regression-test/data/ddl_p0/test_ctas.out | 8 +-
.../data/index_p0/test_bitmap_index.out | 92 +-
.../test_decimal_bitmap_index_multi_page.out | 2 +-
.../data/load_p0/broker_load/simple_array.csv | 3 +-
.../data/load_p0/broker_load/test_array_load.out | 4 +
.../data/load_p0/stream_load/test_stream_load.out | 4 +-
.../json_function/test_query_json_array.out | 15 +
.../json_function/test_query_json_object.out | 20 +-
.../data/query_p0/aggregate/aggregate.out | 3 +
.../cast_function/test_cast_string_to_array.out | 9 +-
.../suites/correctness/test_backends_table.groovy | 23 +-
.../table_valued_function/test_numbers.groovy | 134 +-
regression-test/suites/ddl_p0/test_ctas.groovy | 19 +-
.../http_rest_api/post/test_query_stmt.groovy | 2 +-
.../suites/index_p0/test_index_meta.groovy | 167 ++
..._object.groovy => test_query_json_array.groovy} | 31 +-
.../json_function/test_query_json_object.groovy | 27 +-
.../suites/query_p0/aggregate/aggregate.groovy | 2 +
.../datetime_functions/test_date_function.groovy | 2 +-
.../rollup/test_materialized_view_date.groovy | 8 +-
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q1.groovy | 77 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q10.groovy | 102 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q11.groovy | 91 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q12.groovy | 93 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q13.groovy | 77 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q14.groovy | 63 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q15.groovy | 94 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q16.groovy | 97 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q17.groovy | 73 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q18.groovy | 108 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q19.groovy | 106 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q2.groovy | 129 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q20.groovy | 99 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy | 107 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q22.groovy | 110 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q3.groovy | 83 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q4.groovy | 77 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q5.groovy | 87 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q6.groovy | 54 +
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q7.groovy | 115 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q8.groovy | 112 ++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q9.groovy | 95 +
tools/ssb-tools/ssb-queries/q4.1.sql | 4 +-
tools/ssb-tools/ssb-queries/q4.2.sql | 4 +-
tools/ssb-tools/ssb-queries/q4.3.sql | 4 +-
186 files changed, 6557 insertions(+), 5143 deletions(-)
create mode 100644 be/src/exec/schema_scanner/schema_backends_scanner.cpp
rename be/src/{vec/exec/ves_http_scanner.h => exec/schema_scanner/schema_backends_scanner.h} (52%)
create mode 100644 be/src/olap/inverted_index_parser.cpp
copy be/src/olap/{rowset/segment_v2/options.h => inverted_index_parser.h} (50%)
create mode 100644 be/src/vec/common/hash_table/join_hash_map.h
create mode 100644 be/src/vec/common/hash_table/join_hash_table.h
delete mode 100644 be/src/vec/exec/ves_http_scan_node.cpp
delete mode 100644 be/src/vec/exec/ves_http_scan_node.h
delete mode 100644 be/src/vec/exec/ves_http_scanner.cpp
delete mode 100644 be/src/vec/exec/vjdbc_scan_node.cpp
delete mode 100644 be/src/vec/exec/vjdbc_scan_node.h
delete mode 100644 be/src/vec/exec/vodbc_scan_node.cpp
delete mode 100644 be/src/vec/exec/vodbc_scan_node.h
delete mode 100644 be/src/vec/exec/volap_scan_node.cpp
delete mode 100644 be/src/vec/exec/volap_scan_node.h
delete mode 100644 be/src/vec/exec/volap_scanner.cpp
delete mode 100644 be/src/vec/exec/volap_scanner.h
create mode 100644 be/test/vec/columns/column_decimal_test.cpp
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/VisitorRewriteJob.java
rename fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/{ResolveAggregateFunctions.java => FillUpMissingSlots.java} (86%)
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PullUpPredicates.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/statistics/OlapTableStats.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/statistics/TabletStats.java
rename fe/fe-core/src/test/java/org/apache/doris/nereids/parser/{ResolveAggregateFunctionsTest.java => FillUpMissingSlotsTest.java} (99%)
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicatesTest.java
create mode 100644 regression-test/data/query/sql_functions/json_function/test_query_json_array.out
copy fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/JobType.java => regression-test/suites/correctness/test_backends_table.groovy (73%)
create mode 100644 regression-test/suites/index_p0/test_index_meta.groovy
copy regression-test/suites/query/sql_functions/json_function/{test_query_json_object.groovy => test_query_json_array.groovy} (53%)
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q1.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q10.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q11.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q12.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q13.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q14.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q15.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q16.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q17.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q18.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q19.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q2.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q20.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q22.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q3.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q4.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q5.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q6.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q7.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q8.groovy
create mode 100644 regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q9.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 22/39: [docs](odbc) fix docs for sqlserver odbc table (#14017)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 44df5b384a088ac4243fe00741407e9e3aee2839
Author: xueweizhang <zx...@163.com>
AuthorDate: Wed Nov 9 08:39:39 2022 +0800
[docs](odbc) fix docs for sqlserver odbc table (#14017)
Signed-off-by: nextdreamblue <zx...@163.com>
Signed-off-by: nextdreamblue <zx...@163.com>
---
.../docs/ecosystem/external-table/odbc-of-doris.md | 26 ++++++++++++++++++---
.../docs/ecosystem/external-table/odbc-of-doris.md | 27 +++++++++++++++++++---
2 files changed, 47 insertions(+), 6 deletions(-)
diff --git a/docs/en/docs/ecosystem/external-table/odbc-of-doris.md b/docs/en/docs/ecosystem/external-table/odbc-of-doris.md
index 661f56ec35..816a955343 100644
--- a/docs/en/docs/ecosystem/external-table/odbc-of-doris.md
+++ b/docs/en/docs/ecosystem/external-table/odbc-of-doris.md
@@ -106,14 +106,15 @@ The following parameters are accepted by ODBC external table:
Parameter | Description
---|---
**hosts** | IP address of external database
+**port** | Port of external database
**driver** | The driver name of ODBC Driver, which needs to be/conf/odbcinst.ini. The driver names should be consistent.
**type** | The type of external database, currently supports Oracle, MySQL and PostgerSQL
**user** | The user name of database
**password** | password for the user
-**charset** | charset of connection
+**charset** | charset of connection(it is useless for sqlserver)
Remark:
-In addition to adding the above parameters to `PROPERTIES`, you can also add parameters specific to each database's ODBC driver implementation, such as `sslverify` for mysql, etc.
+In addition to adding the above parameters to `PROPERTIES`, you can also add parameters specific to each database's ODBC driver implementation, such as `sslverify` for mysql、`ClientCharset` for sqlserver, etc.
>Notes:
>
@@ -203,7 +204,7 @@ This contains all versions of PostgreSQL from 9.x to 14.x, including the corresp
####
-| Oracle版本 | Oracle ODBC版本 |
+| Oracle version | Oracle ODBC version |
| ------------------------------------------------------------ | ------------------------------------------ |
| Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production | oracle-instantclient19.13-odbc-19.13.0.0.0 |
| Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production | oracle-instantclient19.13-odbc-19.13.0.0.0 |
@@ -220,6 +221,14 @@ https://download.oracle.com/otn_software/linux/instantclient/1913000/oracle-inst
https://download.oracle.com/otn_software/linux/instantclient/1913000/oracle-instantclient19.13-basic-19.13.0.0.0-2.x86_64.rpm
```
+#### 4.SQLServer
+
+| SQLServer version | SQLServer ODBC version |
+| --------- | -------------- |
+| SQL Server 2016 Enterprise | freetds-1.2.21 |
+
+Currently only tested this version, other versions will be added after testing
+
## Ubuntu operating system
The unixODBC versions used are: 2.3.4, Doris 0.15, Ubuntu 20.04
@@ -259,6 +268,14 @@ sudo alien -i oracle-instantclient19.13-odbc-19.13.0.0.0-2.x86_64.rpm
sudo alien -i oracle-instantclient19.13-sqlplus-19.13.0.0.0-2.x86_64.rpm
````
+#### 4.SQLServer
+
+| SQLServer version | SQLServer ODBC version |
+| --------- | -------------- |
+| SQL Server 2016 Enterprise | freetds-1.2.21 |
+
+Currently only tested this version, other versions will be added after testing
+
## Data type mapping
There are different data types among different databases. Here, the types in each database and the data type matching in Doris are listed.
@@ -389,3 +406,6 @@ This is the compatibility problem between MySQL database ODBC driver and existin
The default encoding used by Doris when connecting to odbc tables is utf8, since the default utf8 encoding in mysql is utf8mb3, it can't represent the emoji expressions which need 4-byte encoding. Here need to set `charset`=`utf8mb4` when you create odbc mysql tables, then can read and write emoji normally 😀.
+12. Set charset for sqlserver odbc table
+
+ It is useless to set `charset` parameter for sqlserver odbc table,you can set `ClientCharset` parameter (for freetds) to get write or read data, for example: "ClientCharset" = "UTF-8".
\ No newline at end of file
diff --git a/docs/zh-CN/docs/ecosystem/external-table/odbc-of-doris.md b/docs/zh-CN/docs/ecosystem/external-table/odbc-of-doris.md
index e895062f94..f5c1e94756 100644
--- a/docs/zh-CN/docs/ecosystem/external-table/odbc-of-doris.md
+++ b/docs/zh-CN/docs/ecosystem/external-table/odbc-of-doris.md
@@ -102,18 +102,19 @@ PROPERTIES (
参数 | 说明
---|---
**hosts** | 外表数据库的IP地址
+**port** | 外表数据库的服务端口号
**driver** | ODBC外表的Driver名,该名字需要和be/conf/odbcinst.ini中的Driver名一致。
**odbc_type** | 外表数据库的类型,当前支持oracle, mysql, postgresql
**user** | 外表数据库的用户名
**password** | 对应用户的密码信息
-**charset** | 数据库连接使用的字符集
+**charset** | 数据库连接使用的字符集(对sqlserver无效)
备注:
-`PROPERTIES` 中除了可以添加上述参数之外,还支持每个数据库的ODBC driver 实现的专用参数,比如mysql 的`sslverify` 等
+`PROPERTIES` 中除了可以添加上述参数之外,还支持每个数据库的ODBC driver 实现的专用参数,比如mysql 的`sslverify`、sqlserver 的`ClientCharset`等
>注意:
>
->如果你是SQL Server 2017 及之后的版本,因为SQL Server 2017及之后版本默认开启了安全认证,你需要再定义 ODBC Resources的时候加上 `"TrustServerCertificate"="Yes"`
+>如果你是SQL Server 2017 及之后的版本,因为SQL Server 2017及之后版本默认开启了安全认证,你需要在定义 ODBC Resources的时候加上 `"TrustServerCertificate"="Yes"`
##### ODBC Driver的安装和配置
@@ -214,6 +215,14 @@ https://download.oracle.com/otn_software/linux/instantclient/1913000/oracle-inst
https://download.oracle.com/otn_software/linux/instantclient/1913000/oracle-instantclient19.13-basic-19.13.0.0.0-2.x86_64.rpm
```
+#### 4.SQLServer
+
+| SQLServer版本 | SQLServer ODBC版本 |
+| --------- | -------------- |
+| SQL Server 2016 Enterprise | freetds-1.2.21 |
+
+目前只测试了这一个版本其他版本测试后补充
+
### Ubuntu操作系统
使用的unixODBC版本是:2.3.4,Doris 0.15,Ubuntu 20.04
@@ -253,6 +262,14 @@ sudo alien -i oracle-instantclient19.13-odbc-19.13.0.0.0-2.x86_64.rpm
sudo alien -i oracle-instantclient19.13-sqlplus-19.13.0.0.0-2.x86_64.rpm
```
+#### 4.SQLServer
+
+| SQLServer版本 | SQLServer ODBC版本 |
+| --------- | -------------- |
+| SQL Server 2016 Enterprise | freetds-1.2.21 |
+
+目前只测试了这一个版本其他版本测试后补充
+
## 类型匹配
@@ -377,3 +394,7 @@ sudo alien -i oracle-instantclient19.13-sqlplus-19.13.0.0.0-2.x86_64.rpm
11. 读写mysql外表的emoji表情出现乱码
Doris进行odbc外表连接时,默认采用的编码为utf8,由于mysql之中默认的utf8编码为utf8mb3,无法表示需要4字节编码的emoji表情。这里需要在建立mysql外表时设置`charset`=`utf8mb4`,便可以正常读写emoji表情😀。
+
+12. 读写sqlserver外表的编码配置
+
+ 由于sqlserver的odbc外表连接时,无法直接通过`charset`来配置编码,用户可以使用`ClientCharset`(for freetds)配置项来设置, 比如 "ClientCharset" = "UTF-8"。
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 11/39: [enhancement](Nereids) tpch q21 anti and semi join reorder (#14037)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6d699fbcb5a00f7c86085eb816d33298dec10587
Author: minghong <mi...@163.com>
AuthorDate: Tue Nov 8 17:21:50 2022 +0800
[enhancement](Nereids) tpch q21 anti and semi join reorder (#14037)
estimation of anti and semi join need re-work. we just let tpch q21 pass.
---
.../apache/doris/nereids/stats/JoinEstimation.java | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java
index dc5345e891..afe6bdf3a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java
@@ -147,16 +147,33 @@ public class JoinEstimation {
return result;
}
+ private static double estimateLeftSemiJoin(double leftCount, double rightCount) {
+ return leftCount - leftCount / Math.max(2, rightCount);
+ }
+
/**
* estimate join
*/
public static StatsDeriveResult estimateV2(StatsDeriveResult leftStats, StatsDeriveResult rightStats, Join join) {
JoinType joinType = join.getJoinType();
double rowCount = Double.MAX_VALUE;
+ //TODO the estimation of semi and anti join is not proper, just for tpch q21
if (joinType == JoinType.LEFT_SEMI_JOIN || joinType == JoinType.LEFT_ANTI_JOIN) {
- rowCount = leftStats.getRowCount();
+ double rightCount = rightStats.getRowCount();
+ double leftCount = leftStats.getRowCount();
+ if (join.getHashJoinConjuncts().isEmpty()) {
+ rowCount = joinType == JoinType.LEFT_SEMI_JOIN ? leftCount : 0;
+ } else {
+ rowCount = estimateLeftSemiJoin(leftCount, rightCount);
+ }
} else if (joinType == JoinType.RIGHT_SEMI_JOIN || joinType == JoinType.RIGHT_ANTI_JOIN) {
- rowCount = rightStats.getRowCount();
+ double rightCount = rightStats.getRowCount();
+ double leftCount = leftStats.getRowCount();
+ if (join.getHashJoinConjuncts().isEmpty()) {
+ rowCount = joinType == JoinType.RIGHT_SEMI_JOIN ? rightCount : 0;
+ } else {
+ rowCount = estimateLeftSemiJoin(rightCount, leftCount);
+ }
} else if (joinType == JoinType.INNER_JOIN) {
if (join.getHashJoinConjuncts().isEmpty()) {
//TODO: consider other join conjuncts
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 19/39: [feature](inverted index)WIP inverted index api: SQL syntax and metadata (#13430)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c84ab7970547057e61fbc2e500ea4883b590b359
Author: Kang <kx...@gmail.com>
AuthorDate: Tue Nov 8 23:46:53 2022 +0800
[feature](inverted index)WIP inverted index api: SQL syntax and metadata (#13430)
Introduce a SQL syntax for creating inverted index and related metadata changes.
```
-- create table with INVERTED index
CREATE TABLE httplogs (
ts datetime,
clientip varchar(20),
request string,
status smallint,
size int,
INDEX idx_size (size) USING INVERTED,
INDEX idx_status (status) USING INVERTED,
INDEX idx_clientip (clientip) USING INVERTED PROPERTIES("parser"="none")
)
DUPLICATE KEY(ts)
DISTRIBUTED BY RANDOM BUCKETS 10
-- add an INVERTED index to a table
CREATE INDEX idx_request ON httplogs(request) USING INVERTED PROPERTIES("parser"="english");
```
---
be/src/olap/CMakeLists.txt | 1 +
be/src/olap/inverted_index_parser.cpp | 65 ++++++++
be/src/olap/inverted_index_parser.h | 46 ++++++
be/src/olap/tablet_meta.cpp | 35 +++++
be/src/olap/tablet_schema.cpp | 120 +++++++++++++++
be/src/olap/tablet_schema.h | 29 ++++
fe/fe-core/src/main/cup/sql_parser.cup | 15 +-
.../apache/doris/analysis/CreateIndexClause.java | 6 +-
.../org/apache/doris/analysis/CreateTableStmt.java | 5 +-
.../java/org/apache/doris/analysis/IndexDef.java | 52 ++++++-
.../apache/doris/analysis/InvertedIndexUtil.java | 56 +++++++
.../org/apache/doris/analysis/ShowIndexStmt.java | 1 +
.../main/java/org/apache/doris/catalog/Index.java | 56 ++++++-
.../java/org/apache/doris/qe/ShowExecutor.java | 2 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
.../doris/analysis/CreateIndexClauseTest.java | 2 +-
.../org/apache/doris/analysis/IndexDefTest.java | 6 +-
.../org/apache/doris/catalog/OlapTableTest.java | 4 +-
.../persist/TableAddOrDropColumnsInfoTest.java | 2 +-
gensrc/proto/olap_file.proto | 15 ++
gensrc/thrift/Descriptors.thrift | 6 +-
.../data/index_p0/test_bitmap_index.out | 92 ++++++------
.../test_decimal_bitmap_index_multi_page.out | 2 +-
.../suites/index_p0/test_index_meta.groovy | 167 +++++++++++++++++++++
24 files changed, 710 insertions(+), 76 deletions(-)
diff --git a/be/src/olap/CMakeLists.txt b/be/src/olap/CMakeLists.txt
index 225f60dad2..f46dd6259d 100644
--- a/be/src/olap/CMakeLists.txt
+++ b/be/src/olap/CMakeLists.txt
@@ -41,6 +41,7 @@ add_library(Olap STATIC
file_helper.cpp
generic_iterators.cpp
hll.cpp
+ inverted_index_parser.cpp
bloom_filter_predicate.cpp
like_column_predicate.cpp
key_coder.cpp
diff --git a/be/src/olap/inverted_index_parser.cpp b/be/src/olap/inverted_index_parser.cpp
new file mode 100644
index 0000000000..9407b52ee4
--- /dev/null
+++ b/be/src/olap/inverted_index_parser.cpp
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/inverted_index_parser.h"
+
+#include "util/string_util.h"
+
+namespace doris {
+
+std::string inverted_index_parser_type_to_string(InvertedIndexParserType parser_type) {
+ switch (parser_type) {
+ case InvertedIndexParserType::PARSER_NONE:
+ return INVERTED_INDEX_PARSER_NONE;
+ case InvertedIndexParserType::PARSER_STANDARD:
+ return INVERTED_INDEX_PARSER_STANDARD;
+ case InvertedIndexParserType::PARSER_ENGLISH:
+ return INVERTED_INDEX_PARSER_ENGLISH;
+ case InvertedIndexParserType::PARSER_CHINESE:
+ return INVERTED_INDEX_PARSER_CHINESE;
+ default:
+ return INVERTED_INDEX_PARSER_UNKNOWN;
+ }
+
+ return INVERTED_INDEX_PARSER_UNKNOWN;
+}
+
+InvertedIndexParserType get_inverted_index_parser_type_from_string(const std::string& parser_str) {
+ auto parser_str_lower = to_lower(parser_str);
+ if (parser_str_lower == INVERTED_INDEX_PARSER_NONE) {
+ return InvertedIndexParserType::PARSER_NONE;
+ } else if (parser_str_lower == INVERTED_INDEX_PARSER_STANDARD) {
+ return InvertedIndexParserType::PARSER_STANDARD;
+ } else if (parser_str_lower == INVERTED_INDEX_PARSER_ENGLISH) {
+ return InvertedIndexParserType::PARSER_ENGLISH;
+ } else if (parser_str_lower == INVERTED_INDEX_PARSER_CHINESE) {
+ return InvertedIndexParserType::PARSER_CHINESE;
+ }
+
+ return InvertedIndexParserType::PARSER_UNKNOWN;
+}
+
+std::string get_parser_string_from_properties(
+ const std::map<std::string, std::string>& properties) {
+ if (properties.find(INVERTED_INDEX_PARSER_KEY) != properties.end()) {
+ return properties.at(INVERTED_INDEX_PARSER_KEY);
+ } else {
+ return INVERTED_INDEX_PARSER_NONE;
+ }
+}
+
+} // namespace doris
diff --git a/be/src/olap/inverted_index_parser.h b/be/src/olap/inverted_index_parser.h
new file mode 100644
index 0000000000..0c870aa355
--- /dev/null
+++ b/be/src/olap/inverted_index_parser.h
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <map>
+#include <string>
+
+namespace doris {
+
+enum class InvertedIndexParserType {
+ PARSER_UNKNOWN = 0,
+ PARSER_NONE = 1,
+ PARSER_STANDARD = 2,
+ PARSER_ENGLISH = 3,
+ PARSER_CHINESE = 4,
+};
+
+const std::string INVERTED_INDEX_PARSER_KEY = "parser";
+const std::string INVERTED_INDEX_PARSER_UNKNOWN = "unknown";
+const std::string INVERTED_INDEX_PARSER_NONE = "none";
+const std::string INVERTED_INDEX_PARSER_STANDARD = "standard";
+const std::string INVERTED_INDEX_PARSER_ENGLISH = "english";
+const std::string INVERTED_INDEX_PARSER_CHINESE = "chinese";
+
+std::string inverted_index_parser_type_to_string(InvertedIndexParserType parser_type);
+
+InvertedIndexParserType get_inverted_index_parser_type_from_string(const std::string& parser_str);
+
+std::string get_parser_string_from_properties(const std::map<std::string, std::string>& properties);
+
+} // namespace doris
diff --git a/be/src/olap/tablet_meta.cpp b/be/src/olap/tablet_meta.cpp
index 70348ba110..010b2217c6 100644
--- a/be/src/olap/tablet_meta.cpp
+++ b/be/src/olap/tablet_meta.cpp
@@ -169,6 +169,41 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t partition_id, int64_t tablet_id
}
}
+ // copy index meta
+ if (tablet_schema.__isset.indexes) {
+ for (auto& index : tablet_schema.indexes) {
+ TabletIndexPB* index_pb = schema->add_index();
+ index_pb->set_index_id(index.index_id);
+ index_pb->set_index_name(index.index_name);
+ // init col_unique_id in index at be side, since col_unique_id may be -1 at fe side
+ // get column unique id by name
+ for (auto column_name : index.columns) {
+ for (auto column : schema->column()) {
+ if (iequal(column.name(), column_name)) {
+ index_pb->add_col_unique_id(column.unique_id());
+ }
+ }
+ }
+ switch (index.index_type) {
+ case TIndexType::BITMAP:
+ index_pb->set_index_type(IndexType::BITMAP);
+ break;
+ case TIndexType::INVERTED:
+ index_pb->set_index_type(IndexType::INVERTED);
+ break;
+ case TIndexType::BLOOMFILTER:
+ index_pb->set_index_type(IndexType::BLOOMFILTER);
+ break;
+ }
+ if (index.__isset.properties) {
+ auto properties = index_pb->mutable_properties();
+ for (auto kv : index.properties) {
+ (*properties)[kv.first] = kv.second;
+ }
+ }
+ }
+ }
+
schema->set_next_column_unique_id(next_unique_id);
if (has_bf_columns && tablet_schema.__isset.bloom_filter_fpp) {
schema->set_bf_fpp(tablet_schema.bloom_filter_fpp);
diff --git a/be/src/olap/tablet_schema.cpp b/be/src/olap/tablet_schema.cpp
index 9dcc7a5793..fe4b353e5a 100644
--- a/be/src/olap/tablet_schema.cpp
+++ b/be/src/olap/tablet_schema.cpp
@@ -469,6 +469,62 @@ vectorized::AggregateFunctionPtr TabletColumn::get_aggregate_function(
agg_name, argument_types, {}, argument_types.back()->is_nullable());
}
+void TabletIndex::init_from_thrift(const TOlapTableIndex& index,
+ const TabletSchema& tablet_schema) {
+ _index_id = index.index_id;
+ _index_name = index.index_name;
+ // init col_unique_id in index at be side, since col_unique_id may be -1 at fe side
+ // get column unique id by name
+ std::vector<int32_t> col_unique_ids(index.columns.size());
+ for (size_t i = 0; i < index.columns.size(); i++) {
+ col_unique_ids[i] = tablet_schema.column(index.columns[i]).unique_id();
+ }
+ _col_unique_ids = std::move(col_unique_ids);
+
+ switch (index.index_type) {
+ case TIndexType::BITMAP:
+ _index_type = IndexType::BITMAP;
+ break;
+ case TIndexType::INVERTED:
+ _index_type = IndexType::INVERTED;
+ break;
+ case TIndexType::BLOOMFILTER:
+ _index_type = IndexType::BLOOMFILTER;
+ break;
+ }
+ if (index.__isset.properties) {
+ for (auto kv : index.properties) {
+ _properties[kv.first] = kv.second;
+ }
+ }
+}
+
+void TabletIndex::init_from_pb(const TabletIndexPB& index) {
+ _index_id = index.index_id();
+ _index_name = index.index_name();
+ _col_unique_ids.clear();
+ for (auto col_unique_id : index.col_unique_id()) {
+ _col_unique_ids.push_back(col_unique_id);
+ }
+ _index_type = index.index_type();
+ for (auto& kv : index.properties()) {
+ _properties[kv.first] = kv.second;
+ }
+}
+
+void TabletIndex::to_schema_pb(TabletIndexPB* index) const {
+ index->set_index_id(_index_id);
+ index->set_index_name(_index_name);
+ index->clear_col_unique_id();
+ for (auto col_unique_id : _col_unique_ids) {
+ index->add_col_unique_id(col_unique_id);
+ }
+ index->set_index_type(_index_type);
+ for (auto& kv : _properties) {
+ (*index->mutable_properties())[kv.first] = kv.second;
+ }
+}
+
void TabletSchema::append_column(TabletColumn column, bool is_dropped_column) {
if (column.is_key()) {
_num_key_columns++;
@@ -522,6 +578,11 @@ void TabletSchema::init_from_pb(const TabletSchemaPB& schema) {
_cols.emplace_back(std::move(column));
_num_columns++;
}
+ for (auto& index_pb : schema.index()) {
+ TabletIndex index;
+ index.init_from_pb(index_pb);
+ _indexes.emplace_back(std::move(index));
+ }
_num_short_key_columns = schema.num_short_key_columns();
_num_rows_per_row_block = schema.num_rows_per_row_block();
_compress_kind = schema.compress_kind();
@@ -647,6 +708,10 @@ void TabletSchema::to_schema_pb(TabletSchemaPB* tablet_schema_pb) const {
ColumnPB* column = tablet_schema_pb->add_column();
col.to_schema_pb(column);
}
+ for (auto& index : _indexes) {
+ auto index_pb = tablet_schema_pb->add_index();
+ index.to_schema_pb(index_pb);
+ }
tablet_schema_pb->set_num_short_key_columns(_num_short_key_columns);
tablet_schema_pb->set_num_rows_per_row_block(_num_rows_per_row_block);
tablet_schema_pb->set_compress_kind(_compress_kind);
@@ -710,11 +775,66 @@ const TabletColumn& TabletSchema::column_by_uid(int32_t col_unique_id) const {
return _cols.at(_field_id_to_index.at(col_unique_id));
}
+void TabletSchema::update_indexes_from_thrift(const std::vector<doris::TOlapTableIndex>& tindexes) {
+ std::vector<TabletIndex> indexes;
+ for (auto& tindex : tindexes) {
+ TabletIndex index;
+ index.init_from_thrift(tindex, *this);
+ indexes.emplace_back(std::move(index));
+ }
+ _indexes = std::move(indexes);
+}
+
const TabletColumn& TabletSchema::column(const std::string& field_name) const {
const auto& found = _field_name_to_index.find(field_name);
return _cols[found->second];
}
+std::vector<const TabletIndex*> TabletSchema::get_indexes_for_column(int32_t col_unique_id) const {
+ std::vector<const TabletIndex*> indexes_for_column;
+
+ // TODO use more efficient impl
+ for (size_t i = 0; i < _indexes.size(); i++) {
+ for (int32_t id : _indexes[i].col_unique_ids()) {
+ if (id == col_unique_id) {
+ indexes_for_column.push_back(&(_indexes[i]));
+ }
+ }
+ }
+
+ return indexes_for_column;
+}
+
+bool TabletSchema::has_inverted_index(int32_t col_unique_id) const {
+ // TODO use more efficient impl
+ for (size_t i = 0; i < _indexes.size(); i++) {
+ if (_indexes[i].index_type() == IndexType::INVERTED) {
+ for (int32_t id : _indexes[i].col_unique_ids()) {
+ if (id == col_unique_id) {
+ return true;
+ }
+ }
+ }
+ }
+
+ return false;
+}
+
+const TabletIndex* TabletSchema::get_inverted_index(int32_t col_unique_id) const {
+ // TODO use more efficient impl
+ for (size_t i = 0; i < _indexes.size(); i++) {
+ if (_indexes[i].index_type() == IndexType::INVERTED) {
+ for (int32_t id : _indexes[i].col_unique_ids()) {
+ if (id == col_unique_id) {
+ return &(_indexes[i]);
+ }
+ }
+ }
+ }
+
+ return nullptr;
+}
+
vectorized::Block TabletSchema::create_block(
const std::vector<uint32_t>& return_columns,
const std::unordered_set<uint32_t>* tablet_columns_need_convert_null) const {
diff --git a/be/src/olap/tablet_schema.h b/be/src/olap/tablet_schema.h
index 300b603e83..6702901e83 100644
--- a/be/src/olap/tablet_schema.h
+++ b/be/src/olap/tablet_schema.h
@@ -117,6 +117,28 @@ private:
bool operator==(const TabletColumn& a, const TabletColumn& b);
bool operator!=(const TabletColumn& a, const TabletColumn& b);
+class TabletSchema;
+
+class TabletIndex {
+public:
+ void init_from_thrift(const TOlapTableIndex& index, const TabletSchema& tablet_schema);
+ void init_from_pb(const TabletIndexPB& index);
+ void to_schema_pb(TabletIndexPB* index) const;
+
+ const int64_t index_id() const { return _index_id; }
+ const std::string& index_name() const { return _index_name; }
+ const IndexType index_type() const { return _index_type; }
+ const vector<int32_t>& col_unique_ids() const { return _col_unique_ids; }
+ const std::map<string, string>& properties() const { return _properties; }
+
+private:
+ int64_t _index_id;
+ std::string _index_name;
+ IndexType _index_type;
+ std::vector<int32_t> _col_unique_ids;
+ std::map<string, string> _properties;
+};
+
class TabletSchema {
public:
// TODO(yingchun): better to make constructor as private to avoid
@@ -161,6 +183,12 @@ public:
int32_t sequence_col_idx() const { return _sequence_col_idx; }
segment_v2::CompressionTypePB compression_type() const { return _compression_type; }
+ const std::vector<TabletIndex>& indexes() const { return _indexes; }
+ std::vector<const TabletIndex*> get_indexes_for_column(int32_t col_unique_id) const;
+ bool has_inverted_index(int32_t col_unique_id) const;
+ const TabletIndex* get_inverted_index(int32_t col_unique_id) const;
+ void update_indexes_from_thrift(const std::vector<doris::TOlapTableIndex>& indexes);
+
int32_t schema_version() const { return _schema_version; }
void clear_columns();
vectorized::Block create_block(
@@ -196,6 +224,7 @@ private:
SortType _sort_type = SortType::LEXICAL;
size_t _sort_col_num = 0;
std::vector<TabletColumn> _cols;
+ std::vector<TabletIndex> _indexes;
std::unordered_map<std::string, int32_t> _field_name_to_index;
std::unordered_map<int32_t, int32_t> _field_id_to_index;
size_t _num_columns = 0;
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup b/fe/fe-core/src/main/cup/sql_parser.cup
index a9cf6b3dea..adec23db2c 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -409,6 +409,7 @@ terminal String
KW_IS,
KW_ISNULL,
KW_ISOLATION,
+ KW_INVERTED,
KW_JOB,
KW_JOIN,
KW_JSON,
@@ -1830,9 +1831,9 @@ create_stmt ::=
{:
RESULT = new CreateMultiTableMaterializedViewStmt(mvName, buildMethod, refreshInfo, keyDesc, partitionDesc, distributionDesc, properties, query);
:}
- | KW_CREATE KW_INDEX opt_if_not_exists:ifNotExists ident:indexName KW_ON table_name:tableName LPAREN ident_list:cols RPAREN opt_index_type:indexType opt_comment:comment
+ | KW_CREATE KW_INDEX opt_if_not_exists:ifNotExists ident:indexName KW_ON table_name:tableName LPAREN ident_list:cols RPAREN opt_index_type:indexType opt_properties:properties opt_comment:comment
{:
- RESULT = new AlterTableStmt(tableName, Lists.newArrayList(new CreateIndexClause(tableName, new IndexDef(indexName, ifNotExists, cols, indexType, comment), false)));
+ RESULT = new AlterTableStmt(tableName, Lists.newArrayList(new CreateIndexClause(tableName, new IndexDef(indexName, ifNotExists, cols, indexType, properties, comment), false)));
:}
/* resource */
| KW_CREATE opt_external:isExternal KW_RESOURCE opt_if_not_exists:ifNotExists ident_or_text:resourceName opt_properties:properties
@@ -3142,9 +3143,9 @@ column_definition ::=
;
index_definition ::=
- KW_INDEX opt_if_not_exists:ifNotExists ident:indexName LPAREN ident_list:cols RPAREN opt_index_type:indexType opt_comment:comment
+ KW_INDEX opt_if_not_exists:ifNotExists ident:indexName LPAREN ident_list:cols RPAREN opt_index_type:indexType opt_properties:properties opt_comment:comment
{:
- RESULT = new IndexDef(indexName, ifNotExists, cols, indexType, comment);
+ RESULT = new IndexDef(indexName, ifNotExists, cols, indexType, properties, comment);
:}
;
@@ -3181,6 +3182,10 @@ opt_index_type ::=
{:
RESULT = IndexDef.IndexType.BITMAP;
:}
+ | KW_USING KW_INVERTED
+ {:
+ RESULT = IndexDef.IndexType.INVERTED;
+ :}
;
opt_if_exists ::=
@@ -6489,6 +6494,8 @@ keyword ::=
{: RESULT = id; :}
| KW_INDEXES:id
{: RESULT = id; :}
+ | KW_INVERTED:id
+ {: RESULT = id; :}
| KW_ISNULL:id
{: RESULT = id; :}
| KW_ISOLATION:id
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateIndexClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateIndexClause.java
index d1ff4006ba..632492191c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateIndexClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateIndexClause.java
@@ -18,6 +18,7 @@
package org.apache.doris.analysis;
import org.apache.doris.alter.AlterOpType;
+import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.Index;
import org.apache.doris.common.AnalysisException;
@@ -70,8 +71,9 @@ public class CreateIndexClause extends AlterTableClause {
throw new AnalysisException("index definition expected.");
}
indexDef.analyze();
- this.index = new Index(indexDef.getIndexName(), indexDef.getColumns(), indexDef.getIndexType(),
- indexDef.getComment());
+ this.index = new Index(Env.getCurrentEnv().getNextId(), indexDef.getIndexName(),
+ indexDef.getColumns(), indexDef.getIndexType(),
+ indexDef.getProperties(), indexDef.getComment());
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index 7ba06377a0..7f33428da3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -488,8 +488,9 @@ public class CreateTableStmt extends DdlStmt {
+ indexColName);
}
}
- indexes.add(new Index(indexDef.getIndexName(), indexDef.getColumns(), indexDef.getIndexType(),
- indexDef.getComment()));
+ indexes.add(new Index(Env.getCurrentEnv().getNextId(), indexDef.getIndexName(),
+ indexDef.getColumns(), indexDef.getIndexType(),
+ indexDef.getProperties(), indexDef.getComment()));
distinct.add(indexDef.getIndexName());
distinctCol.add(indexDef.getColumns().stream().map(String::toUpperCase).collect(Collectors.toList()));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java
index fca80403c1..4df18a9879 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/IndexDef.java
@@ -24,7 +24,9 @@ import org.apache.doris.common.AnalysisException;
import com.google.common.base.Strings;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.TreeSet;
public class IndexDef {
@@ -33,8 +35,10 @@ public class IndexDef {
private List<String> columns;
private IndexType indexType;
private String comment;
+ private Map<String, String> properties;
- public IndexDef(String indexName, boolean ifNotExists, List<String> columns, IndexType indexType, String comment) {
+ public IndexDef(String indexName, boolean ifNotExists, List<String> columns, IndexType indexType,
+ Map<String, String> properties, String comment) {
this.indexName = indexName;
this.ifNotExists = ifNotExists;
this.columns = columns;
@@ -48,12 +52,18 @@ public class IndexDef {
} else {
this.comment = comment;
}
+ if (properties == null) {
+ this.properties = new HashMap<>();
+ } else {
+ this.properties = properties;
+ }
}
public void analyze() throws AnalysisException {
- if (indexType == IndexDef.IndexType.BITMAP) {
+ if (indexType == IndexDef.IndexType.BITMAP
+ || indexType == IndexDef.IndexType.INVERTED) {
if (columns == null || columns.size() != 1) {
- throw new AnalysisException("bitmap index can only apply to a single column.");
+ throw new AnalysisException(indexType.toString() + " index can only apply to a single column.");
}
if (Strings.isNullOrEmpty(indexName)) {
throw new AnalysisException("index name cannot be blank.");
@@ -93,6 +103,19 @@ public class IndexDef {
if (indexType != null) {
sb.append(" USING ").append(indexType.toString());
}
+ if (properties != null && properties.size() > 0) {
+ sb.append(" PROPERTIES(");
+ first = true;
+ for (Map.Entry<String, String> e : properties.entrySet()) {
+ if (first) {
+ first = false;
+ } else {
+ sb.append(", ");
+ }
+ sb.append("\"").append(e.getKey()).append("\"=").append("\"").append(e.getValue()).append("\"");
+ }
+ sb.append(")");
+ }
if (comment != null) {
sb.append(" COMMENT '" + comment + "'");
}
@@ -116,6 +139,10 @@ public class IndexDef {
return indexType;
}
+ public Map<String, String> getProperties() {
+ return properties;
+ }
+
public String getComment() {
return comment;
}
@@ -126,29 +153,38 @@ public class IndexDef {
public enum IndexType {
BITMAP,
+ INVERTED,
+ BLOOMFILTER,
+ }
+ public boolean isInvertedIndex() {
+ return (this.indexType == IndexType.INVERTED);
}
public void checkColumn(Column column, KeysType keysType) throws AnalysisException {
- if (indexType == IndexType.BITMAP) {
+ if (indexType == IndexType.BITMAP || indexType == IndexType.INVERTED || indexType == IndexType.BLOOMFILTER) {
String indexColName = column.getName();
PrimitiveType colType = column.getDataType();
if (!(colType.isDateType() || colType.isDecimalV2Type() || colType.isDecimalV3Type()
|| colType.isFixedPointType() || colType.isStringType() || colType == PrimitiveType.BOOLEAN)) {
- throw new AnalysisException(colType + " is not supported in bitmap index. "
+ throw new AnalysisException(colType + " is not supported in " + indexType.toString() + " index. "
+ "invalid column: " + indexColName);
} else if ((keysType == KeysType.AGG_KEYS && !column.isKey())) {
- throw new AnalysisException(
- "BITMAP index only used in columns of DUP_KEYS/UNIQUE_KEYS table or key columns of"
+ throw new AnalysisException(indexType.toString()
+ + " index only used in columns of DUP_KEYS/UNIQUE_KEYS table or key columns of"
+ " AGG_KEYS table. invalid column: " + indexColName);
}
+
+ if (indexType == IndexType.INVERTED) {
+ InvertedIndexUtil.checkInvertedIndexParser(indexColName, colType, properties);
+ }
} else {
throw new AnalysisException("Unsupported index type: " + indexType);
}
}
public void checkColumns(List<Column> columns, KeysType keysType) throws AnalysisException {
- if (indexType == IndexType.BITMAP) {
+ if (indexType == IndexType.BITMAP || indexType == IndexType.INVERTED || indexType == IndexType.BLOOMFILTER) {
for (Column col : columns) {
checkColumn(col, keysType);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java
new file mode 100644
index 0000000000..412935ab91
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.AnalysisException;
+
+import java.util.Map;
+
+public class InvertedIndexUtil {
+
+ public static String INVERTED_INDEX_PARSER_KEY = "parser";
+ public static String INVERTED_INDEX_PARSER_UNKNOWN = "unknown";
+ public static String INVERTED_INDEX_PARSER_NONE = "none";
+ public static String INVERTED_INDEX_PARSER_STANDARD = "standard";
+ public static String INVERTED_INDEX_PARSER_ENGLISH = "english";
+ public static String INVERTED_INDEX_PARSER_CHINESE = "chinese";
+
+ public static String getInvertedIndexParser(Map<String, String> properties) {
+ String parser = properties == null ? null : properties.get(INVERTED_INDEX_PARSER_KEY);
+ // default is "none" if not set
+ return parser != null ? parser : INVERTED_INDEX_PARSER_NONE;
+ }
+
+ public static void checkInvertedIndexParser(String indexColName, PrimitiveType colType,
+ Map<String, String> properties) throws AnalysisException {
+ String parser = getInvertedIndexParser(properties);
+ if (colType.isStringType()) {
+ if (!(parser.equals(INVERTED_INDEX_PARSER_NONE)
+ || parser.equals(INVERTED_INDEX_PARSER_STANDARD)
+ || parser.equals(INVERTED_INDEX_PARSER_ENGLISH)
+ || parser.equals(INVERTED_INDEX_PARSER_CHINESE))) {
+ throw new AnalysisException("INVERTED index parser: " + parser
+ + " is invalid for column: " + indexColName + " of type " + colType);
+ }
+ } else if (!parser.equals(INVERTED_INDEX_PARSER_NONE)) {
+ throw new AnalysisException("INVERTED index with parser: " + parser
+ + " is not supported for column: " + indexColName + " of type " + colType);
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java
index 38468e5c7c..0a3977f03f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java
@@ -46,6 +46,7 @@ public class ShowIndexStmt extends ShowStmt {
.addColumn(new Column("Null", ScalarType.createVarchar(80)))
.addColumn(new Column("Index_type", ScalarType.createVarchar(80)))
.addColumn(new Column("Comment", ScalarType.createVarchar(80)))
+ .addColumn(new Column("Properties", ScalarType.createVarchar(200)))
.build();
private String dbName;
private TableName tableName;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
index b5536faf55..2da8ce35ff 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java
@@ -18,8 +18,10 @@
package org.apache.doris.catalog;
import org.apache.doris.analysis.IndexDef;
+import org.apache.doris.analysis.InvertedIndexUtil;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
+import org.apache.doris.common.util.PrintableMap;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.thrift.TIndexType;
import org.apache.doris.thrift.TOlapTableIndex;
@@ -30,26 +32,37 @@ import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
/**
* Internal representation of index, including index type, name, columns and comments.
* This class will used in olaptable
*/
public class Index implements Writable {
+ public static final int INDEX_ID_INIT_VALUE = -1;
+
+ @SerializedName(value = "indexId")
+ private long indexId = -1; // -1 for compatibale
@SerializedName(value = "indexName")
private String indexName;
@SerializedName(value = "columns")
private List<String> columns;
@SerializedName(value = "indexType")
private IndexDef.IndexType indexType;
+ @SerializedName(value = "properties")
+ private Map<String, String> properties;
@SerializedName(value = "comment")
private String comment;
- public Index(String indexName, List<String> columns, IndexDef.IndexType indexType, String comment) {
+ public Index(long indexId, String indexName, List<String> columns,
+ IndexDef.IndexType indexType, Map<String, String> properties, String comment) {
+ this.indexId = indexId;
this.indexName = indexName;
this.columns = columns;
this.indexType = indexType;
+ this.properties = properties;
this.comment = comment;
}
@@ -57,9 +70,18 @@ public class Index implements Writable {
this.indexName = null;
this.columns = null;
this.indexType = null;
+ this.properties = null;
this.comment = null;
}
+ public long getIndexId() {
+ return indexId;
+ }
+
+ public void setIndexId(long indexId) {
+ this.indexId = indexId;
+ }
+
public String getIndexName() {
return indexName;
}
@@ -84,6 +106,26 @@ public class Index implements Writable {
this.indexType = indexType;
}
+ public Map<String, String> getProperties() {
+ return properties;
+ }
+
+ public void setProperties(Map<String, String> properties) {
+ this.properties = properties;
+ }
+
+ public String getPropertiesString() {
+ if (properties == null || properties.isEmpty()) {
+ return "";
+ }
+
+ return "(" + new PrintableMap(properties, "=", true, false, ",").toString() + ")";
+ }
+
+ public String getInvertedIndexParser() {
+ return InvertedIndexUtil.getInvertedIndexParser(properties);
+ }
+
public String getComment() {
return comment;
}
@@ -108,7 +150,8 @@ public class Index implements Writable {
}
public Index clone() {
- return new Index(indexName, new ArrayList<>(columns), indexType, comment);
+ return new Index(indexId, indexName, new ArrayList<>(columns),
+ indexType, new HashMap<>(properties), comment);
}
@Override
@@ -133,6 +176,10 @@ public class Index implements Writable {
if (indexType != null) {
sb.append(" USING ").append(indexType.toString());
}
+ if (properties != null && properties.size() > 0) {
+ sb.append(" PROPERTIES");
+ sb.append(getPropertiesString());
+ }
if (comment != null) {
sb.append(" COMMENT '" + comment + "'");
}
@@ -141,11 +188,12 @@ public class Index implements Writable {
public TOlapTableIndex toThrift() {
TOlapTableIndex tIndex = new TOlapTableIndex();
+ tIndex.setIndexId(indexId);
tIndex.setIndexName(indexName);
tIndex.setColumns(columns);
tIndex.setIndexType(TIndexType.valueOf(indexType.toString()));
- if (columns != null) {
- tIndex.setComment(comment);
+ if (properties != null) {
+ tIndex.setProperties(properties);
}
return tIndex;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index 0c7077a84f..f72aab6f0c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -963,7 +963,7 @@ public class ShowExecutor {
for (Index index : indexes) {
rows.add(Lists.newArrayList(showStmt.getTableName().toString(), "", index.getIndexName(),
"", String.join(",", index.getColumns()), "", "", "", "",
- "", index.getIndexType().name(), index.getComment()));
+ "", index.getIndexType().name(), index.getComment(), index.getPropertiesString()));
}
} finally {
table.readUnlock();
diff --git a/fe/fe-core/src/main/jflex/sql_scanner.flex b/fe/fe-core/src/main/jflex/sql_scanner.flex
index 4a15fe3521..541322b57a 100644
--- a/fe/fe-core/src/main/jflex/sql_scanner.flex
+++ b/fe/fe-core/src/main/jflex/sql_scanner.flex
@@ -118,6 +118,7 @@ import org.apache.doris.qe.SqlModeHelper;
keywordMap.put("bin", new Integer(SqlParserSymbols.KW_BIN));
keywordMap.put("binlog", new Integer(SqlParserSymbols.KW_BINLOG));
keywordMap.put("bitmap", new Integer(SqlParserSymbols.KW_BITMAP));
+ keywordMap.put("inverted", new Integer(SqlParserSymbols.KW_INVERTED));
keywordMap.put("bitmap_union", new Integer(SqlParserSymbols.KW_BITMAP_UNION));
keywordMap.put("blob", new Integer(SqlParserSymbols.KW_BLOB));
keywordMap.put("boolean", new Integer(SqlParserSymbols.KW_BOOLEAN));
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateIndexClauseTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateIndexClauseTest.java
index fe46c0c4af..88aae9c4e2 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateIndexClauseTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateIndexClauseTest.java
@@ -37,7 +37,7 @@ public class CreateIndexClauseTest {
public void testNormal() throws AnalysisException {
CreateIndexClause clause = new CreateIndexClause(
new TableName(InternalCatalog.INTERNAL_CATALOG_NAME, "db", "table"),
- new IndexDef("index1", false, Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, "balabala"),
+ new IndexDef("index1", false, Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, null, "balabala"),
false);
clause.analyze(analyzer);
Assert.assertEquals("CREATE INDEX index1 ON `db`.`table` (`col1`) USING BITMAP COMMENT 'balabala'",
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/IndexDefTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/IndexDefTest.java
index 5f18742cb9..b1072e41fa 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/IndexDefTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/IndexDefTest.java
@@ -29,7 +29,7 @@ public class IndexDefTest {
@Before
public void setUp() throws Exception {
- def = new IndexDef("index1", false, Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, "balabala");
+ def = new IndexDef("index1", false, Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, null, "balabala");
}
@Test
@@ -46,7 +46,7 @@ public class IndexDefTest {
+ "x1xxxxxxxxxxxxxxxxxindex1xxxxxxxxxxxxxxxxxindex1xxxxxxxxxxxxxxxxxindex1xxxxxxxxxxxxx"
+ "xxxxindex1xxxxxxxxxxxxxxxxxindex1xxxxxxxxxxxxxxxxxindex1xxxxxxxxxxxxxxxxxindex1xxxxx"
+ "xxxxxxxxxxxxindex1xxxxxxxxxxxxxxxxx", false,
- Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP,
+ Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, null,
"balabala");
def.analyze();
Assert.fail("No exception throws.");
@@ -54,7 +54,7 @@ public class IndexDefTest {
Assert.assertTrue(e instanceof AnalysisException);
}
try {
- def = new IndexDef("", false, Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, "balabala");
+ def = new IndexDef("", false, Lists.newArrayList("col1"), IndexDef.IndexType.BITMAP, null, "balabala");
def.analyze();
Assert.fail("No exception throws.");
} catch (AnalysisException e) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
index 871389d6dd..f547189ff1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
@@ -56,8 +56,8 @@ public class OlapTableTest {
continue;
}
OlapTable tbl = (OlapTable) table;
- tbl.setIndexes(Lists.newArrayList(new Index("index", Lists.newArrayList("col"),
- IndexDef.IndexType.BITMAP, "xxxxxx")));
+ tbl.setIndexes(Lists.newArrayList(new Index(0, "index", Lists.newArrayList("col"),
+ IndexDef.IndexType.BITMAP, null, "xxxxxx")));
System.out.println("orig table id: " + tbl.getId());
FastByteArrayOutputStream byteArrayOutputStream = new FastByteArrayOutputStream();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/persist/TableAddOrDropColumnsInfoTest.java b/fe/fe-core/src/test/java/org/apache/doris/persist/TableAddOrDropColumnsInfoTest.java
index a97dd27bf9..e849198a5b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/persist/TableAddOrDropColumnsInfoTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/persist/TableAddOrDropColumnsInfoTest.java
@@ -65,7 +65,7 @@ public class TableAddOrDropColumnsInfoTest {
indexSchemaMap.put(tableId, fullSchema);
List<Index> indexes = Lists.newArrayList(
- new Index("index", Lists.newArrayList("testCol1"), IndexDef.IndexType.BITMAP, "xxxxxx"));
+ new Index(0, "index", Lists.newArrayList("testCol1"), IndexDef.IndexType.BITMAP, null, "xxxxxx"));
TableAddOrDropColumnsInfo tableAddOrDropColumnsInfo1 = new TableAddOrDropColumnsInfo(dbId, tableId,
indexSchemaMap, indexes, jobId);
diff --git a/gensrc/proto/olap_file.proto b/gensrc/proto/olap_file.proto
index f19228b935..30b127fc78 100644
--- a/gensrc/proto/olap_file.proto
+++ b/gensrc/proto/olap_file.proto
@@ -201,6 +201,20 @@ message ColumnPB {
repeated string children_column_names = 18;
}
+enum IndexType {
+ BITMAP = 0;
+ INVERTED = 1;
+ BLOOMFILTER = 2;
+}
+
+message TabletIndexPB {
+ optional int64 index_id = 1;
+ optional string index_name = 2;
+ optional IndexType index_type = 3;
+ repeated int32 col_unique_id = 4;
+ map<string, string> properties = 5;
+}
+
enum SortType {
LEXICAL = 0;
ZORDER = 1;
@@ -222,6 +236,7 @@ message TabletSchemaPB {
optional segment_v2.CompressionTypePB compression_type = 13 [default=LZ4F];
optional int32 schema_version = 14;
optional bool disable_auto_compaction = 15 [default=false];
+ repeated TabletIndexPB index = 16;
}
enum TabletStatePB {
diff --git a/gensrc/thrift/Descriptors.thrift b/gensrc/thrift/Descriptors.thrift
index a3fc84571b..d133b3a27d 100644
--- a/gensrc/thrift/Descriptors.thrift
+++ b/gensrc/thrift/Descriptors.thrift
@@ -118,7 +118,9 @@ enum THdfsCompression {
}
enum TIndexType {
- BITMAP
+ BITMAP,
+ INVERTED,
+ BLOOMFILTER
}
// Mapping from names defined by Avro to the enum.
@@ -195,6 +197,8 @@ struct TOlapTableIndex {
2: optional list<string> columns
3: optional TIndexType index_type
4: optional string comment
+ 5: optional i64 index_id
+ 6: optional map<string, string> properties
}
struct TTabletLocation {
diff --git a/regression-test/data/index_p0/test_bitmap_index.out b/regression-test/data/index_p0/test_bitmap_index.out
index 3b37588e29..ff0754c309 100644
--- a/regression-test/data/index_p0/test_bitmap_index.out
+++ b/regression-test/data/index_p0/test_bitmap_index.out
@@ -17,21 +17,21 @@ k14 DATETIMEV2(3) Yes false \N NONE
k15 DATETIMEV2(6) Yes false \N NONE
-- !sql --
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index1 k1 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index2 k2 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index3 k3 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index4 k4 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index5 k5 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index6 k6 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index7 k7 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index8 k8 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index9 k9 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index10 k10 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index11 k11 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index12 k12 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index13 k13 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index14 k14 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_dup index15 k15 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index1 k1 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index2 k2 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index3 k3 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index4 k4 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index5 k5 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index6 k6 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index7 k7 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index8 k8 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index9 k9 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index10 k10 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index11 k11 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index12 k12 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index13 k13 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index14 k14 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_dup index15 k15 BITMAP
-- !sql --
1 1 1 1 1 1 2022-05-31 2022-05-31T10:00 1 1 true 2022-05-31 2022-05-31T10:00 2022-05-31T10:00:00.111 2022-05-31T10:00:00.111111
@@ -55,21 +55,21 @@ k15 DATETIMEV2(6) Yes true \N
v1 INT Yes false \N SUM
-- !sql --
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index1 k1 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index2 k2 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index3 k3 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index4 k4 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index5 k5 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index6 k6 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index7 k7 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index8 k8 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index9 k9 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index10 k10 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index11 k11 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index12 k12 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index13 k13 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index14 k14 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_agg index15 k15 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index1 k1 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index2 k2 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index3 k3 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index4 k4 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index5 k5 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index6 k6 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index7 k7 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index8 k8 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index9 k9 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index10 k10 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index11 k11 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index12 k12 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index13 k13 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index14 k14 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_agg index15 k15 BITMAP
-- !sql --
1 1 1 1 1 1 2022-05-31 2022-05-31T10:00 1 1 true 2022-05-31 2022-05-31T10:00 2022-05-31T10:00:00.111 2022-05-31T10:00:00.111111 1
@@ -93,22 +93,22 @@ k15 DATETIMEV2(6) Yes false \N REPLACE
v1 INT Yes false \N REPLACE
-- !sql --
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index1 k1 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index2 k2 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index3 k3 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index4 k4 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index5 k5 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index6 k6 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index7 k7 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index8 k8 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index9 k9 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index10 k10 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index11 k11 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index12 k12 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index13 k13 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index14 k14 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index15 k15 BITMAP
-default_cluster:regression_test_index_p0.test_bitmap_index_unique index16 v1 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index1 k1 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index2 k2 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index3 k3 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index4 k4 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index5 k5 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index6 k6 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index7 k7 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index8 k8 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index9 k9 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index10 k10 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index11 k11 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index12 k12 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index13 k13 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index14 k14 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index15 k15 BITMAP
+default_cluster:regression_test_index_p0.test_bitmap_index_unique index16 v1 BITMAP
-- !sql --
1 1 1 1 1 1 2022-05-31 2022-05-31T10:00 1 1 true 2022-05-31 2022-05-31T10:00 2022-05-31T10:00:00.111 2022-05-31T10:00:00.111111 1
diff --git a/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out b/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out
index 538cbb9404..49b5a9da49 100644
--- a/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out
+++ b/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out
@@ -3,7 +3,7 @@
a DECIMAL(12,6) No true \N
-- !sql --
-default_cluster:regression_test_index_p0.test_decimal_bitmap_index_multi_page bitmap_index_multi_page a BITMAP
+default_cluster:regression_test_index_p0.test_decimal_bitmap_index_multi_page bitmap_index_multi_page a BITMAP
-- !sql --
0.000001
diff --git a/regression-test/suites/index_p0/test_index_meta.groovy b/regression-test/suites/index_p0/test_index_meta.groovy
new file mode 100644
index 0000000000..f31c125220
--- /dev/null
+++ b/regression-test/suites/index_p0/test_index_meta.groovy
@@ -0,0 +1,167 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import groovy.json.JsonOutput
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite("index_meta", "p0") {
+ // prepare test table
+ def timeout = 60000
+ def delta_time = 1000
+ def alter_res = "null"
+ def useTime = 0
+ def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+ for(int t = delta_time; t <= OpTimeout; t += delta_time){
+ alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName = "${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+ alter_res = alter_res.toString()
+ if(alter_res.contains("FINISHED")) {
+ break
+ }
+ useTime = t
+ sleep(delta_time)
+ }
+ assertTrue(useTime <= OpTimeout)
+ }
+
+ def tableName = "test_index_meta"
+
+ sql "DROP TABLE IF EXISTS ${tableName}"
+ // create 1 replica table
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName} (
+ `id` INT NULL,
+ `name` STRING NULL,
+ `description` STRING NULL,
+ INDEX idx_id (`id`) USING BITMAP COMMENT 'index for id',
+ INDEX idx_name (`name`) USING INVERTED PROPERTIES("parser"="none") COMMENT 'index for name'
+ )
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ properties("replication_num" = "1");
+ """
+
+ // set enable_vectorized_engine=true
+ sql """ SET enable_vectorized_engine=true; """
+ def var_result = sql "show variables"
+ logger.info("show variales result: " + var_result )
+
+ // show index of create table
+ def show_result = sql "show index from ${tableName}"
+ logger.info("show index from " + tableName + " result: " + show_result)
+ assertEquals(show_result.size(), 2)
+ assertEquals(show_result[0][2], "idx_id")
+ assertEquals(show_result[0][4], "id")
+ assertEquals(show_result[0][10], "BITMAP")
+ assertEquals(show_result[0][11], "index for id")
+ assertEquals(show_result[0][12], "")
+ assertEquals(show_result[1][2], "idx_name")
+ assertEquals(show_result[1][4], "name")
+ assertEquals(show_result[1][10], "INVERTED")
+ assertEquals(show_result[1][11], "index for name")
+ assertEquals(show_result[1][12], "(\"parser\" = \"none\")")
+
+ // add index on column description
+ sql "create index idx_desc on ${tableName}(description) USING INVERTED PROPERTIES(\"parser\"=\"standard\") COMMENT 'index for description';"
+ wait_for_latest_op_on_table_finish(tableName, timeout)
+
+ // show index after add index
+ show_result = sql "show index from ${tableName}"
+ logger.info("show index from " + tableName + " result: " + show_result)
+ assertEquals(show_result.size(), 3)
+ assertEquals(show_result[0][2], "idx_id")
+ assertEquals(show_result[0][4], "id")
+ assertEquals(show_result[0][10], "BITMAP")
+ assertEquals(show_result[0][11], "index for id")
+ assertEquals(show_result[0][12], "")
+ assertEquals(show_result[1][2], "idx_name")
+ assertEquals(show_result[1][4], "name")
+ assertEquals(show_result[1][10], "INVERTED")
+ assertEquals(show_result[1][11], "index for name")
+ assertEquals(show_result[1][12], "(\"parser\" = \"none\")")
+ assertEquals(show_result[2][2], "idx_desc")
+ assertEquals(show_result[2][4], "description")
+ assertEquals(show_result[2][10], "INVERTED")
+ assertEquals(show_result[2][11], "index for description")
+ assertEquals(show_result[2][12], "(\"parser\" = \"standard\")")
+
+ // drop index
+ // add index on column description
+ sql "drop index idx_name on ${tableName}"
+ wait_for_latest_op_on_table_finish(tableName, timeout)
+
+ show_result = sql "show index from ${tableName}"
+ logger.info("show index from " + tableName + " result: " + show_result)
+ assertEquals(show_result.size(), 2)
+ assertEquals(show_result[0][2], "idx_id")
+ assertEquals(show_result[0][4], "id")
+ assertEquals(show_result[0][10], "BITMAP")
+ assertEquals(show_result[0][11], "index for id")
+ assertEquals(show_result[0][12], "")
+ assertEquals(show_result[1][2], "idx_desc")
+ assertEquals(show_result[1][4], "description")
+ assertEquals(show_result[1][10], "INVERTED")
+ assertEquals(show_result[1][11], "index for description")
+ assertEquals(show_result[1][12], "(\"parser\" = \"standard\")")
+
+ // add index on column description
+ sql "create index idx_name on ${tableName}(name) USING INVERTED COMMENT 'new index for name';"
+ wait_for_latest_op_on_table_finish(tableName, timeout)
+
+ // show index after add index
+ show_result = sql "show index from ${tableName}"
+ logger.info("show index from " + tableName + " result: " + show_result)
+ assertEquals(show_result.size(), 3)
+ assertEquals(show_result[0][2], "idx_id")
+ assertEquals(show_result[0][4], "id")
+ assertEquals(show_result[0][10], "BITMAP")
+ assertEquals(show_result[0][11], "index for id")
+ assertEquals(show_result[0][12], "")
+ assertEquals(show_result[1][2], "idx_desc")
+ assertEquals(show_result[1][4], "description")
+ assertEquals(show_result[1][10], "INVERTED")
+ assertEquals(show_result[1][11], "index for description")
+ assertEquals(show_result[1][12], "(\"parser\" = \"standard\")")
+ assertEquals(show_result[2][2], "idx_name")
+ assertEquals(show_result[2][4], "name")
+ assertEquals(show_result[2][10], "INVERTED")
+ assertEquals(show_result[2][11], "new index for name")
+ assertEquals(show_result[2][12], "")
+
+
+ def show_tablets_result = sql "show tablets from ${tableName}"
+ logger.info("show tablets from " + tableName + " result: " + show_tablets_result)
+ for (j in range(0, show_tablets_result.size())) {
+ String metaUrl = show_tablets_result[j][16]
+ String getMetaCommand = "curl -X GET " + metaUrl
+ def process = getMetaCommand.toString().execute()
+ int code = process.waitFor()
+ String err = IOGroovyMethods.getText(new BufferedReader(new InputStreamReader(process.getErrorStream())));
+ String out = process.getText()
+ logger.info("get meta process result: code=" + code + ", out=" + out + ", err=" + err)
+ assertEquals(code, 0)
+ def json = parseJson(out.trim())
+ assert json.schema.index instanceof List
+ int i = 0;
+ for (Object index in (List) json.schema.index) {
+ // assertEquals(index.index_id, i);
+ assertEquals(index.index_name, show_result[i][2])
+ assertEquals(index.index_type, show_result[i][10])
+ // assertEquals(index.properties, show_result[j][12]);
+ i++;
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 33/39: [fix](Nereids) aggregate disassemble generate error output list on GLOBAL phase aggregate (#14079)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit dd2e4b04251619c72494807ac8fe064bab85cdf0
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Wed Nov 9 13:43:12 2022 +0800
[fix](Nereids) aggregate disassemble generate error output list on GLOBAL phase aggregate (#14079)
we must use localAggregateFunction as key of globalOutputSMap, because we use local output exprs to generate global output in disassembleDistinct
---
.../rules/rewrite/AggregateDisassemble.java | 9 ++--
.../rewrite/logical/AggregateDisassembleTest.java | 63 ++++++++++++++++++++++
2 files changed, 69 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
index c4e3db8765..4a632e188b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/AggregateDisassemble.java
@@ -188,11 +188,12 @@ public class AggregateDisassemble extends OneRewriteRuleFactory {
continue;
}
- NamedExpression localOutputExpr = new Alias(aggregateFunction.withAggregateParam(
+ AggregateFunction localAggregateFunction = aggregateFunction.withAggregateParam(
aggregateFunction.getAggregateParam()
.withDistinct(false)
.withGlobal(false)
- ), aggregateFunction.toSql());
+ );
+ NamedExpression localOutputExpr = new Alias(localAggregateFunction, aggregateFunction.toSql());
List<DataType> inputTypesBeforeDissemble = aggregateFunction.children()
.stream()
@@ -207,7 +208,9 @@ public class AggregateDisassemble extends OneRewriteRuleFactory {
.withChildren(Lists.newArrayList(localOutputExpr.toSlot()));
inputSubstitutionMap.put(aggregateFunction, substitutionValue);
- globalOutputSMap.put(aggregateFunction, substitutionValue);
+ // because we use local output exprs to generate global output in disassembleDistinct,
+ // so we must use localAggregateFunction as key
+ globalOutputSMap.put(localAggregateFunction, substitutionValue);
localOutputExprs.add(localOutputExpr);
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/AggregateDisassembleTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/AggregateDisassembleTest.java
index 2f3e5303fb..af737bdeb3 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/AggregateDisassembleTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/AggregateDisassembleTest.java
@@ -22,6 +22,7 @@ import org.apache.doris.nereids.trees.expressions.Add;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateParam;
import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
@@ -38,11 +39,13 @@ import org.apache.doris.nereids.util.PlanConstructor;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
+import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import java.util.List;
+import java.util.Optional;
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class AggregateDisassembleTest implements PatternMatchSupported {
@@ -261,4 +264,64 @@ public class AggregateDisassembleTest implements PatternMatchSupported {
0).getExprId())
);
}
+
+ @Test
+ public void distinctWithNormalAggregateFunction() {
+ List<Expression> groupExpressionList = Lists.newArrayList(rStudent.getOutput().get(0).toSlot());
+ List<NamedExpression> outputExpressionList = Lists.newArrayList(
+ new Alias(new Count(AggregateParam.distinctAndGlobal(), rStudent.getOutput().get(2).toSlot()), "c"),
+ new Alias(new Sum(rStudent.getOutput().get(0).toSlot()), "sum"));
+ Plan root = new LogicalAggregate<>(groupExpressionList, outputExpressionList, rStudent);
+
+ // check local:
+ // id
+ Expression localOutput0 = rStudent.getOutput().get(0);
+ // sum
+ Sum localOutput1 = new Sum(new AggregateParam(false, false, Optional.empty()), rStudent.getOutput().get(0).toSlot());
+ // age
+ Expression localOutput2 = rStudent.getOutput().get(2);
+ // id
+ Expression localGroupBy0 = rStudent.getOutput().get(0);
+ // age
+ Expression localGroupBy1 = rStudent.getOutput().get(2);
+
+ PlanChecker.from(MemoTestUtils.createConnectContext(), root)
+ .applyTopDown(new AggregateDisassemble())
+ .matchesFromRoot(
+ logicalAggregate(
+ logicalAggregate(
+ logicalAggregate()
+ .when(agg -> agg.getAggPhase().equals(AggPhase.LOCAL))
+ .when(agg -> agg.getOutputExpressions().get(0).equals(localOutput0))
+ .when(agg -> agg.getOutputExpressions().get(1).child(0).equals(localOutput1))
+ .when(agg -> agg.getOutputExpressions().get(2).equals(localOutput2))
+ .when(agg -> agg.getGroupByExpressions().get(0).equals(localGroupBy0))
+ .when(agg -> agg.getGroupByExpressions().get(1).equals(localGroupBy1))
+ ).when(agg -> agg.getAggPhase().equals(AggPhase.GLOBAL))
+ .when(agg -> agg.getOutputExpressions().get(0)
+ .equals(agg.child().getOutputExpressions().get(0)))
+ .when(agg -> {
+ Slot child = agg.child().getOutputExpressions().get(1).toSlot();
+ Assertions.assertTrue(agg.getOutputExpressions().get(1).child(0) instanceof Sum);
+ return ((Sum) agg.getOutputExpressions().get(1).child(0)).child().equals(child);
+ })
+ .when(agg -> agg.getOutputExpressions().get(2)
+ .equals(agg.child().getOutputExpressions().get(2)))
+ .when(agg -> agg.getGroupByExpressions().get(0)
+ .equals(agg.child().getOutputExpressions().get(0)))
+ .when(agg -> agg.getGroupByExpressions().get(1)
+ .equals(agg.child().getOutputExpressions().get(2)))
+ ).when(agg -> agg.getAggPhase().equals(AggPhase.DISTINCT_LOCAL))
+ .when(agg -> agg.getOutputExpressions().size() == 2)
+ .when(agg -> agg.getOutputExpressions().get(0) instanceof Alias)
+ .when(agg -> agg.getOutputExpressions().get(0).child(0) instanceof Count)
+ .when(agg -> agg.getOutputExpressions().get(1).child(0) instanceof Sum)
+ .when(agg -> agg.getOutputExpressions().get(0).getExprId() == outputExpressionList.get(
+ 0).getExprId())
+ .when(agg -> agg.getOutputExpressions().get(1).getExprId() == outputExpressionList.get(
+ 1).getExprId())
+ .when(agg -> agg.getGroupByExpressions().get(0)
+ .equals(agg.child().child().getOutputExpressions().get(0)))
+ );
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 10/39: [regression](Nereids) add back tpch regression test cases (#13826)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit dd2a122707e52952a365ac160c180aacfbfc8a93
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Tue Nov 8 16:40:46 2022 +0800
[regression](Nereids) add back tpch regression test cases (#13826)
1. add back TPC-H regression test cases
2. fix decimal problem on aggregate function sum and agg introduced by #13764
3. fix memo merge group NPE introduced by #13900
---
.../java/org/apache/doris/nereids/memo/Memo.java | 14 ++-
.../trees/expressions/functions/agg/Avg.java | 2 +-
.../trees/expressions/functions/agg/Sum.java | 4 +-
.../apache/doris/nereids/types/DecimalType.java | 4 +-
.../nereids/trees/expressions/GetDataTypeTest.java | 2 +-
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q1.groovy | 77 ++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q10.groovy | 102 ++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q11.groovy | 91 +++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q12.groovy | 93 +++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q13.groovy | 77 ++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q14.groovy | 63 ++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q15.groovy | 94 +++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q16.groovy | 97 ++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q17.groovy | 73 ++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q18.groovy | 108 +++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q19.groovy | 106 +++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q2.groovy | 129 +++++++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q20.groovy | 99 ++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy | 107 +++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q22.groovy | 110 ++++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q3.groovy | 83 +++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q4.groovy | 77 ++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q5.groovy | 87 ++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q6.groovy | 54 +++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q7.groovy | 115 ++++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q8.groovy | 112 ++++++++++++++++++
.../suites/tpch_sf1_p1/tpch_sf1/nereids/q9.groovy | 95 +++++++++++++++
27 files changed, 2064 insertions(+), 11 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
index 7db42fba56..81b1dc7c3f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
@@ -375,11 +375,17 @@ public class Memo {
if (source.equals(destination)) {
return source;
}
- if (source.getParentGroupExpressions().stream()
- .anyMatch(e -> e.getOwnerGroup().equals(destination))) {
- return null;
+ List<GroupExpression> needReplaceChild = Lists.newArrayList();
+ for (GroupExpression groupExpression : groupExpressions.values()) {
+ if (groupExpression.children().contains(source)) {
+ if (groupExpression.getOwnerGroup().equals(destination)) {
+ // cycle, we should not merge
+ return null;
+ }
+ needReplaceChild.add(groupExpression);
+ }
}
- for (GroupExpression groupExpression : source.getParentGroupExpressions()) {
+ for (GroupExpression groupExpression : needReplaceChild) {
groupExpressions.remove(groupExpression);
List<Group> children = groupExpression.children();
// TODO: use a better way to replace child, avoid traversing all groupExpression
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Avg.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Avg.java
index 406004236d..842724458c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Avg.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Avg.java
@@ -58,7 +58,7 @@ public class Avg extends AggregateFunction implements UnaryExpression, ImplicitC
.map(types -> types.get(0))
.orElse(child().getDataType());
if (argumentType instanceof DecimalType) {
- return argumentType;
+ return DecimalType.SYSTEM_DEFAULT;
} else if (argumentType.isDate()) {
return DateType.INSTANCE;
} else if (argumentType.isDateTime()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum.java
index 997bf0757e..f69a1461c1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum.java
@@ -56,12 +56,10 @@ public class Sum extends AggregateFunction implements UnaryExpression, ImplicitC
if (dataType instanceof LargeIntType) {
return dataType;
} else if (dataType instanceof DecimalType) {
- // TODO: precision + 10
- return dataType;
+ return DecimalType.SYSTEM_DEFAULT;
} else if (dataType instanceof IntegralType) {
return BigIntType.INSTANCE;
} else if (dataType instanceof FractionalType) {
- // TODO: precision + 10
return DoubleType.INSTANCE;
} else {
throw new IllegalStateException("Unsupported sum type: " + dataType);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalType.java
index e19b3ee52f..077d1780ac 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DecimalType.java
@@ -37,7 +37,7 @@ public class DecimalType extends FractionalType {
public static int MAX_PRECISION = 38;
public static int MAX_SCALE = 38;
- public static final DecimalType SYSTEM_DEFAULT = new DecimalType(MAX_PRECISION, 18);
+ public static final DecimalType SYSTEM_DEFAULT = new DecimalType(27, 9);
private static final DecimalType BOOLEAN_DECIMAL = new DecimalType(1, 0);
private static final DecimalType TINYINT_DECIMAL = new DecimalType(3, 0);
@@ -100,7 +100,7 @@ public class DecimalType extends FractionalType {
@Override
public Type toCatalogDataType() {
- return ScalarType.createDecimalType(precision, scale);
+ return ScalarType.createDecimalType(27, 9);
}
public int getPrecision() {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/GetDataTypeTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/GetDataTypeTest.java
index 0abf8db6dd..87e5e2115f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/GetDataTypeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/GetDataTypeTest.java
@@ -73,7 +73,7 @@ public class GetDataTypeTest {
Assertions.assertEquals(LargeIntType.INSTANCE, new Sum(largeIntLiteral).getDataType());
Assertions.assertEquals(DoubleType.INSTANCE, new Sum(floatLiteral).getDataType());
Assertions.assertEquals(DoubleType.INSTANCE, new Sum(doubleLiteral).getDataType());
- Assertions.assertEquals(DecimalType.createDecimalType(BigDecimal.ONE), new Sum(decimalLiteral).getDataType());
+ Assertions.assertEquals(DecimalType.createDecimalType(27, 9), new Sum(decimalLiteral).getDataType());
Assertions.assertEquals(BigIntType.INSTANCE, new Sum(bigIntLiteral).getDataType());
Assertions.assertThrows(RuntimeException.class, () -> new Sum(charLiteral).getDataType());
Assertions.assertThrows(RuntimeException.class, () -> new Sum(varcharLiteral).getDataType());
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q1.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q1.groovy
new file mode 100644
index 0000000000..0964eda946
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q1.groovy
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q1_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ l_returnflag,
+ l_linestatus,
+ sum(l_quantity) as sum_qty,
+ sum(l_extendedprice) as sum_base_price,
+ sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,
+ sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,
+ avg(l_quantity) as avg_qty,
+ avg(l_extendedprice) as avg_price,
+ avg(l_discount) as avg_disc,
+ count(*) as count_order
+ from
+ lineitem
+ where
+ l_shipdate <= date '1998-12-01' - interval '90' day
+ group by
+ l_returnflag,
+ l_linestatus
+ order by
+ l_returnflag,
+ l_linestatus;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=false) */
+ l_returnflag,
+ l_linestatus,
+ sum(l_quantity) as sum_qty,
+ sum(l_extendedprice) as sum_base_price,
+ sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,
+ sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,
+ avg(l_quantity) as avg_qty,
+ avg(l_extendedprice) as avg_price,
+ avg(l_discount) as avg_disc,
+ count(*) as count_order
+ from
+ lineitem
+ where
+ l_shipdate <= date '1998-12-01' - interval '90' day
+ group by
+ l_returnflag,
+ l_linestatus
+ order by
+ l_returnflag,
+ l_linestatus;
+ """
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q10.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q10.groovy
new file mode 100644
index 0000000000..b0b0eb5271
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q10.groovy
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q10_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ c_custkey,
+ c_name,
+ sum(l_extendedprice * (1 - l_discount)) as revenue,
+ c_acctbal,
+ n_name,
+ c_address,
+ c_phone,
+ c_comment
+ from
+ customer,
+ orders,
+ lineitem,
+ nation
+ where
+ c_custkey = o_custkey
+ and l_orderkey = o_orderkey
+ and o_orderdate >= date '1993-10-01'
+ and o_orderdate < date '1993-10-01' + interval '3' month
+ and l_returnflag = 'R'
+ and c_nationkey = n_nationkey
+ group by
+ c_custkey,
+ c_name,
+ c_acctbal,
+ c_phone,
+ n_name,
+ c_address,
+ c_comment
+ order by
+ revenue desc
+ limit 20;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */
+ c_custkey,
+ c_name,
+ sum(t1.l_extendedprice * (1 - t1.l_discount)) as revenue,
+ c_acctbal,
+ n_name,
+ c_address,
+ c_phone,
+ c_comment
+ from
+ customer,
+ (
+ select o_custkey,l_extendedprice,l_discount from lineitem, orders
+ where l_orderkey = o_orderkey
+ and o_orderdate >= date '1993-10-01'
+ and o_orderdate < date '1993-10-01' + interval '3' month
+ and l_returnflag = 'R'
+ ) t1,
+ nation
+ where
+ c_custkey = t1.o_custkey
+ and c_nationkey = n_nationkey
+ group by
+ c_custkey,
+ c_name,
+ c_acctbal,
+ c_phone,
+ n_name,
+ c_address,
+ c_comment
+ order by
+ revenue desc
+ limit 20;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q11.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q11.groovy
new file mode 100644
index 0000000000..86c12a2a07
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q11.groovy
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q11_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ ps_partkey,
+ sum(ps_supplycost * ps_availqty) as value
+ from
+ partsupp,
+ supplier,
+ nation
+ where
+ ps_suppkey = s_suppkey
+ and s_nationkey = n_nationkey
+ and n_name = 'GERMANY'
+ group by
+ ps_partkey having
+ sum(ps_supplycost * ps_availqty) > (
+ select
+ sum(ps_supplycost * ps_availqty) * 0.0001000000
+ from
+ partsupp,
+ supplier,
+ nation
+ where
+ ps_suppkey = s_suppkey
+ and s_nationkey = n_nationkey
+ and n_name = 'GERMANY'
+ )
+ order by
+ value desc, ps_partkey;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */
+ ps_partkey,
+ sum(ps_supplycost * ps_availqty) as value
+ from
+ partsupp,
+ (
+ select s_suppkey
+ from supplier, nation
+ where s_nationkey = n_nationkey and n_name = 'GERMANY'
+ ) B
+ where
+ ps_suppkey = B.s_suppkey
+ group by
+ ps_partkey having
+ sum(ps_supplycost * ps_availqty) > (
+ select
+ sum(ps_supplycost * ps_availqty) * 0.000002
+ from
+ partsupp,
+ (select s_suppkey
+ from supplier, nation
+ where s_nationkey = n_nationkey and n_name = 'GERMANY'
+ ) A
+ where
+ ps_suppkey = A.s_suppkey
+ )
+ order by
+ value desc, ps_partkey;
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q12.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q12.groovy
new file mode 100644
index 0000000000..b6634e3652
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q12.groovy
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q12_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ l_shipmode,
+ sum(case
+ when o_orderpriority = '1-URGENT'
+ or o_orderpriority = '2-HIGH'
+ then 1
+ else 0
+ end) as high_line_count,
+ sum(case
+ when o_orderpriority <> '1-URGENT'
+ and o_orderpriority <> '2-HIGH'
+ then 1
+ else 0
+ end) as low_line_count
+ from
+ orders,
+ lineitem
+ where
+ o_orderkey = l_orderkey
+ and l_shipmode in ('MAIL', 'SHIP')
+ and l_commitdate < l_receiptdate
+ and l_shipdate < l_commitdate
+ and l_receiptdate >= date '1994-01-01'
+ and l_receiptdate < date '1994-01-01' + interval '1' year
+ group by
+ l_shipmode
+ order by
+ l_shipmode;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */
+ l_shipmode,
+ sum(case
+ when o_orderpriority = '1-URGENT'
+ or o_orderpriority = '2-HIGH'
+ then 1
+ else 0
+ end) as high_line_count,
+ sum(case
+ when o_orderpriority <> '1-URGENT'
+ and o_orderpriority <> '2-HIGH'
+ then 1
+ else 0
+ end) as low_line_count
+ from
+ orders,
+ lineitem
+ where
+ o_orderkey = l_orderkey
+ and l_shipmode in ('MAIL', 'SHIP')
+ and l_commitdate < l_receiptdate
+ and l_shipdate < l_commitdate
+ and l_receiptdate >= date '1994-01-01'
+ and l_receiptdate < date '1994-01-01' + interval '1' year
+ group by
+ l_shipmode
+ order by
+ l_shipmode;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q13.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q13.groovy
new file mode 100644
index 0000000000..a11937aa39
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q13.groovy
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q13_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ c_count,
+ count(*) as custdist
+ from
+ (
+ select
+ c_custkey,
+ count(o_orderkey) as c_count
+ from
+ customer left outer join orders on
+ c_custkey = o_custkey
+ and o_comment not like '%special%requests%'
+ group by
+ c_custkey
+ ) as c_orders
+ group by
+ c_count
+ order by
+ custdist desc,
+ c_count desc;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */
+ c_count,
+ count(*) as custdist
+ from
+ (
+ select
+ c_custkey,
+ count(o_orderkey) as c_count
+ from
+ orders right outer join customer on
+ c_custkey = o_custkey
+ and o_comment not like '%special%requests%'
+ group by
+ c_custkey
+ ) as c_orders
+ group by
+ c_count
+ order by
+ custdist desc,
+ c_count desc;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q14.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q14.groovy
new file mode 100644
index 0000000000..40d55cbeaa
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q14.groovy
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q14_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ 100.00 * sum(case
+ when p_type like 'PROMO%'
+ then l_extendedprice * (1 - l_discount)
+ else 0
+ end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue
+from
+ lineitem,
+ part
+where
+ l_partkey = p_partkey
+ and l_shipdate >= date '1995-09-01'
+ and l_shipdate < date '1995-09-01' + interval '1' month;
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */
+ 100.00 * sum(case
+ when p_type like 'PROMO%'
+ then l_extendedprice * (1 - l_discount)
+ else 0
+ end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue
+from
+ part,
+ lineitem
+where
+ l_partkey = p_partkey
+ and l_shipdate >= date '1995-09-01'
+ and l_shipdate < date '1995-09-01' + interval '1' month;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q15.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q15.groovy
new file mode 100644
index 0000000000..eb5c1d3b54
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q15.groovy
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q15_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+
+ sql """
+ drop view if exists revenue0;
+ """
+
+ sql """
+ create view revenue0 (supplier_no, total_revenue) as
+ select
+ l_suppkey,
+ sum(l_extendedprice * (1 - l_discount))
+ from
+ lineitem
+ where
+ l_shipdate >= date '1996-01-01'
+ and l_shipdate < date '1996-01-01' + interval '3' month
+ group by
+ l_suppkey;
+ """
+
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ s_suppkey,
+ s_name,
+ s_address,
+ s_phone,
+ total_revenue
+ from
+ supplier,
+ revenue0
+ where
+ s_suppkey = supplier_no
+ and total_revenue = (
+ select
+ max(total_revenue)
+ from
+ revenue0
+ )
+ order by
+ s_suppkey;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */
+ s_suppkey,
+ s_name,
+ s_address,
+ s_phone,
+ total_revenue
+ from
+ supplier,
+ revenue0
+ where
+ s_suppkey = supplier_no
+ and total_revenue = (
+ select
+ max(total_revenue)
+ from
+ revenue0
+ )
+ order by
+ s_suppkey;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q16.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q16.groovy
new file mode 100644
index 0000000000..1e4a5f7dca
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q16.groovy
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q16_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ p_brand,
+ p_type,
+ p_size,
+ count(distinct ps_suppkey) as supplier_cnt
+from
+ partsupp,
+ part
+where
+ p_partkey = ps_partkey
+ and p_brand <> 'Brand#45'
+ and p_type not like 'MEDIUM POLISHED%'
+ and p_size in (49, 14, 23, 45, 19, 3, 36, 9)
+ and ps_suppkey not in (
+ select
+ s_suppkey
+ from
+ supplier
+ where
+ s_comment like '%Customer%Complaints%'
+ )
+group by
+ p_brand,
+ p_type,
+ p_size
+order by
+ supplier_cnt desc,
+ p_brand,
+ p_type,
+ p_size;
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */
+ p_brand,
+ p_type,
+ p_size,
+ count(distinct ps_suppkey) as supplier_cnt
+from
+ partsupp,
+ part
+where
+ p_partkey = ps_partkey
+ and p_brand <> 'Brand#45'
+ and p_type not like 'MEDIUM POLISHED%'
+ and p_size in (49, 14, 23, 45, 19, 3, 36, 9)
+ and ps_suppkey not in (
+ select
+ s_suppkey
+ from
+ supplier
+ where
+ s_comment like '%Customer%Complaints%'
+ )
+group by
+ p_brand,
+ p_type,
+ p_size
+order by
+ supplier_cnt desc,
+ p_brand,
+ p_type,
+ p_size;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q17.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q17.groovy
new file mode 100644
index 0000000000..f49d433946
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q17.groovy
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q17_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ sum(l_extendedprice) / 7.0 as avg_yearly
+ from
+ lineitem,
+ part
+ where
+ p_partkey = l_partkey
+ and p_brand = 'Brand#23'
+ and p_container = 'MED BOX'
+ and l_quantity < (
+ select
+ 0.2 * avg(l_quantity)
+ from
+ lineitem
+ where
+ l_partkey = p_partkey
+ );
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=1, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */
+ sum(l_extendedprice) / 7.0 as avg_yearly
+ from
+ lineitem join
+ part p1 on p1.p_partkey = l_partkey
+ where
+ p1.p_brand = 'Brand#23'
+ and p1.p_container = 'MED BOX'
+ and l_quantity < (
+ select
+ 0.2 * avg(l_quantity)
+ from
+ lineitem join
+ part p2 on p2.p_partkey = l_partkey
+ where
+ l_partkey = p1.p_partkey
+ and p2.p_brand = 'Brand#23'
+ and p2.p_container = 'MED BOX'
+ );
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q18.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q18.groovy
new file mode 100644
index 0000000000..5e4ede22ac
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q18.groovy
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q18_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ c_name,
+ c_custkey,
+ o_orderkey,
+ o_orderdate,
+ o_totalprice,
+ sum(l_quantity)
+from
+ customer,
+ orders,
+ lineitem
+where
+ o_orderkey in (
+ select
+ l_orderkey
+ from
+ lineitem
+ group by
+ l_orderkey having
+ sum(l_quantity) > 300
+ )
+ and c_custkey = o_custkey
+ and o_orderkey = l_orderkey
+group by
+ c_name,
+ c_custkey,
+ o_orderkey,
+ o_orderdate,
+ o_totalprice
+order by
+ o_totalprice desc,
+ o_orderdate
+limit 100;
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */
+ c_name,
+ c_custkey,
+ t3.o_orderkey,
+ t3.o_orderdate,
+ t3.o_totalprice,
+ sum(t3.l_quantity)
+from
+customer join
+(
+ select * from
+ lineitem join
+ (
+ select * from
+ orders left semi join
+ (
+ select
+ l_orderkey
+ from
+ lineitem
+ group by
+ l_orderkey having sum(l_quantity) > 300
+ ) t1
+ on o_orderkey = t1.l_orderkey
+ ) t2
+ on t2.o_orderkey = l_orderkey
+) t3
+on c_custkey = t3.o_custkey
+group by
+ c_name,
+ c_custkey,
+ t3.o_orderkey,
+ t3.o_orderdate,
+ t3.o_totalprice
+order by
+ t3.o_totalprice desc,
+ t3.o_orderdate
+limit 100;
+
+ """
+}
+
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q19.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q19.groovy
new file mode 100644
index 0000000000..fc31026d98
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q19.groovy
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q19_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ sum(l_extendedprice* (1 - l_discount)) as revenue
+from
+ lineitem,
+ part
+where
+ (
+ p_partkey = l_partkey
+ and p_brand = 'Brand#12'
+ and p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG')
+ and l_quantity >= 1 and l_quantity <= 1 + 10
+ and p_size between 1 and 5
+ and l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ )
+ or
+ (
+ p_partkey = l_partkey
+ and p_brand = 'Brand#23'
+ and p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK')
+ and l_quantity >= 10 and l_quantity <= 10 + 10
+ and p_size between 1 and 10
+ and l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ )
+ or
+ (
+ p_partkey = l_partkey
+ and p_brand = 'Brand#34'
+ and p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG')
+ and l_quantity >= 20 and l_quantity <= 20 + 10
+ and p_size between 1 and 15
+ and l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ );
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=true) */
+ sum(l_extendedprice* (1 - l_discount)) as revenue
+from
+ lineitem,
+ part
+where
+ (
+ p_partkey = l_partkey
+ and p_brand = 'Brand#12'
+ and p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG')
+ and l_quantity >= 1 and l_quantity <= 1 + 10
+ and p_size between 1 and 5
+ and l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ )
+ or
+ (
+ p_partkey = l_partkey
+ and p_brand = 'Brand#23'
+ and p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK')
+ and l_quantity >= 10 and l_quantity <= 10 + 10
+ and p_size between 1 and 10
+ and l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ )
+ or
+ (
+ p_partkey = l_partkey
+ and p_brand = 'Brand#34'
+ and p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG')
+ and l_quantity >= 20 and l_quantity <= 20 + 10
+ and p_size between 1 and 15
+ and l_shipmode in ('AIR', 'AIR REG')
+ and l_shipinstruct = 'DELIVER IN PERSON'
+ );
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q2.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q2.groovy
new file mode 100644
index 0000000000..8628045109
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q2.groovy
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q2_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ s_acctbal,
+ s_name,
+ n_name,
+ p_partkey,
+ p_mfgr,
+ s_address,
+ s_phone,
+ s_comment
+from
+ part,
+ supplier,
+ partsupp,
+ nation,
+ region
+where
+ p_partkey = ps_partkey
+ and s_suppkey = ps_suppkey
+ and p_size = 15
+ and p_type like '%BRASS'
+ and s_nationkey = n_nationkey
+ and n_regionkey = r_regionkey
+ and r_name = 'EUROPE'
+ and ps_supplycost = (
+ select
+ min(ps_supplycost)
+ from
+ partsupp,
+ supplier,
+ nation,
+ region
+ where
+ p_partkey = ps_partkey
+ and s_suppkey = ps_suppkey
+ and s_nationkey = n_nationkey
+ and n_regionkey = r_regionkey
+ and r_name = 'EUROPE'
+ )
+order by
+ s_acctbal desc,
+ n_name,
+ s_name,
+ p_partkey
+limit 100;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=1, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */
+ s_acctbal,
+ s_name,
+ n_name,
+ p_partkey,
+ p_mfgr,
+ s_address,
+ s_phone,
+ s_comment
+from
+ partsupp join
+ (
+ select
+ ps_partkey as a_partkey,
+ min(ps_supplycost) as a_min
+ from
+ partsupp,
+ part,
+ supplier,
+ nation,
+ region
+ where
+ p_partkey = ps_partkey
+ and s_suppkey = ps_suppkey
+ and s_nationkey = n_nationkey
+ and n_regionkey = r_regionkey
+ and r_name = 'EUROPE'
+ and p_size = 15
+ and p_type like '%BRASS'
+ group by ps_partkey
+ ) A on ps_partkey = a_partkey and ps_supplycost=a_min ,
+ part,
+ supplier,
+ nation,
+ region
+where
+ p_partkey = ps_partkey
+ and s_suppkey = ps_suppkey
+ and p_size = 15
+ and p_type like '%BRASS'
+ and s_nationkey = n_nationkey
+ and n_regionkey = r_regionkey
+ and r_name = 'EUROPE'
+
+order by
+ s_acctbal desc,
+ n_name,
+ s_name,
+ p_partkey
+limit 100;
+ """
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q20.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q20.groovy
new file mode 100644
index 0000000000..fa8981b91f
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q20.groovy
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q20_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ s_name,
+ s_address
+from
+ supplier,
+ nation
+where
+ s_suppkey in (
+ select
+ ps_suppkey
+ from
+ partsupp
+ where
+ ps_partkey in (
+ select
+ p_partkey
+ from
+ part
+ where
+ p_name like 'forest%'
+ )
+ and ps_availqty > (
+ select
+ 0.5 * sum(l_quantity)
+ from
+ lineitem
+ where
+ l_partkey = ps_partkey
+ and l_suppkey = ps_suppkey
+ and l_shipdate >= date '1994-01-01'
+ and l_shipdate < date '1994-01-01' + interval '1' year
+ )
+ )
+ and s_nationkey = n_nationkey
+ and n_name = 'CANADA'
+order by
+ s_name;
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */
+s_name, s_address from
+supplier left semi join
+(
+ select * from
+ (
+ select l_partkey,l_suppkey, 0.5 * sum(l_quantity) as l_q
+ from lineitem
+ where l_shipdate >= date '1994-01-01'
+ and l_shipdate < date '1994-01-01' + interval '1' year
+ group by l_partkey,l_suppkey
+ ) t2 join
+ (
+ select ps_partkey, ps_suppkey, ps_availqty
+ from partsupp left semi join part
+ on ps_partkey = p_partkey and p_name like 'forest%'
+ ) t1
+ on t2.l_partkey = t1.ps_partkey and t2.l_suppkey = t1.ps_suppkey
+ and t1.ps_availqty > t2.l_q
+) t3
+on s_suppkey = t3.ps_suppkey
+join nation
+where s_nationkey = n_nationkey
+ and n_name = 'CANADA'
+order by s_name;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy
new file mode 100644
index 0000000000..0407830d15
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q21.groovy
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q21_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ s_name,
+ count(*) as numwait
+from
+ supplier,
+ lineitem l1,
+ orders,
+ nation
+where
+ s_suppkey = l1.l_suppkey
+ and o_orderkey = l1.l_orderkey
+ and o_orderstatus = 'F'
+ and l1.l_receiptdate > l1.l_commitdate
+ and exists (
+ select
+ *
+ from
+ lineitem l2
+ where
+ l2.l_orderkey = l1.l_orderkey
+ and l2.l_suppkey <> l1.l_suppkey
+ )
+ and not exists (
+ select
+ *
+ from
+ lineitem l3
+ where
+ l3.l_orderkey = l1.l_orderkey
+ and l3.l_suppkey <> l1.l_suppkey
+ and l3.l_receiptdate > l3.l_commitdate
+ )
+ and s_nationkey = n_nationkey
+ and n_name = 'SAUDI ARABIA'
+group by
+ s_name
+order by
+ numwait desc,
+ s_name
+limit 100;
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */
+s_name, count(*) as numwait
+from orders join
+(
+ select * from
+ lineitem l2 right semi join
+ (
+ select * from
+ lineitem l3 right anti join
+ (
+ select * from
+ lineitem l1 join
+ (
+ select * from
+ supplier join nation
+ where s_nationkey = n_nationkey
+ and n_name = 'SAUDI ARABIA'
+ ) t1
+ where t1.s_suppkey = l1.l_suppkey and l1.l_receiptdate > l1.l_commitdate
+ ) t2
+ on l3.l_orderkey = t2.l_orderkey and l3.l_suppkey <> t2.l_suppkey and l3.l_receiptdate > l3.l_commitdate
+ ) t3
+ on l2.l_orderkey = t3.l_orderkey and l2.l_suppkey <> t3.l_suppkey
+) t4
+on o_orderkey = t4.l_orderkey and o_orderstatus = 'F'
+group by
+ t4.s_name
+order by
+ numwait desc,
+ t4.s_name
+limit 100;
+ """
+
+}
\ No newline at end of file
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q22.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q22.groovy
new file mode 100644
index 0000000000..bfcd6dd80c
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q22.groovy
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q22_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ cntrycode,
+ count(*) as numcust,
+ sum(c_acctbal) as totacctbal
+from
+ (
+ select
+ substring(c_phone, 1, 2) as cntrycode,
+ c_acctbal
+ from
+ customer
+ where
+ substring(c_phone, 1, 2) in
+ ('13', '31', '23', '29', '30', '18', '17')
+ and c_acctbal > (
+ select
+ avg(c_acctbal)
+ from
+ customer
+ where
+ c_acctbal > 0.00
+ and substring(c_phone, 1, 2) in
+ ('13', '31', '23', '29', '30', '18', '17')
+ )
+ and not exists (
+ select
+ *
+ from
+ orders
+ where
+ o_custkey = c_custkey
+ )
+ ) as custsale
+group by
+ cntrycode
+order by
+ cntrycode;
+ """
+
+ qt_select """
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=16, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */
+ cntrycode,
+ count(*) as numcust,
+ sum(c_acctbal) as totacctbal
+from
+ (
+ select
+ substring(c_phone, 1, 2) as cntrycode,
+ c_acctbal
+ from
+ customer
+ where
+ substring(c_phone, 1, 2) in
+ ('13', '31', '23', '29', '30', '18', '17')
+ and c_acctbal > (
+ select
+ avg(c_acctbal)
+ from
+ customer
+ where
+ c_acctbal > 0.00
+ and substring(c_phone, 1, 2) in
+ ('13', '31', '23', '29', '30', '18', '17')
+ )
+ and not exists (
+ select
+ *
+ from
+ orders
+ where
+ o_custkey = c_custkey
+ )
+ ) as custsale
+group by
+ cntrycode
+order by
+ cntrycode;
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q3.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q3.groovy
new file mode 100644
index 0000000000..0e4c19ab2b
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q3.groovy
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q3_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ l_orderkey,
+ sum(l_extendedprice * (1 - l_discount)) as revenue,
+ o_orderdate,
+ o_shippriority
+ from
+ customer,
+ orders,
+ lineitem
+ where
+ c_mktsegment = 'BUILDING'
+ and c_custkey = o_custkey
+ and l_orderkey = o_orderkey
+ and o_orderdate < date '1995-03-15'
+ and l_shipdate > date '1995-03-15'
+ group by
+ l_orderkey,
+ o_orderdate,
+ o_shippriority
+ order by
+ revenue desc,
+ o_orderdate
+ limit 10;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */
+ l_orderkey,
+ sum(l_extendedprice * (1 - l_discount)) as revenue,
+ o_orderdate,
+ o_shippriority
+ from
+ (
+ select l_orderkey, l_extendedprice, l_discount, o_orderdate, o_shippriority, o_custkey from
+ lineitem join orders
+ where l_orderkey = o_orderkey
+ and o_orderdate < date '1995-03-15'
+ and l_shipdate > date '1995-03-15'
+ ) t1 join customer c
+ on c.c_custkey = t1.o_custkey
+ where c_mktsegment = 'BUILDING'
+ group by
+ l_orderkey,
+ o_orderdate,
+ o_shippriority
+ order by
+ revenue desc,
+ o_orderdate
+ limit 10;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q4.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q4.groovy
new file mode 100644
index 0000000000..3b51baedb4
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q4.groovy
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q4_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ o_orderpriority,
+ count(*) as order_count
+ from
+ orders
+ where
+ o_orderdate >= date '1993-07-01'
+ and o_orderdate < date '1993-07-01' + interval '3' month
+ and exists (
+ select
+ *
+ from
+ lineitem
+ where
+ l_orderkey = o_orderkey
+ and l_commitdate < l_receiptdate
+ )
+ group by
+ o_orderpriority
+ order by
+ o_orderpriority;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=4, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */
+ o_orderpriority,
+ count(*) as order_count
+ from
+ (
+ select
+ *
+ from
+ lineitem
+ where l_commitdate < l_receiptdate
+ ) t1
+ right semi join orders
+ on t1.l_orderkey = o_orderkey
+ where
+ o_orderdate >= date '1993-07-01'
+ and o_orderdate < date '1993-07-01' + interval '3' month
+ group by
+ o_orderpriority
+ order by
+ o_orderpriority;
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q5.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q5.groovy
new file mode 100644
index 0000000000..4fc1e494b9
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q5.groovy
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q5_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+ sql 'set exec_mem_limit=8589934592'
+
+ qt_select """
+ select
+ n_name,
+ sum(l_extendedprice * (1 - l_discount)) as revenue
+ from
+ customer,
+ orders,
+ lineitem,
+ supplier,
+ nation,
+ region
+ where
+ c_custkey = o_custkey
+ and l_orderkey = o_orderkey
+ and l_suppkey = s_suppkey
+ and c_nationkey = s_nationkey
+ and s_nationkey = n_nationkey
+ and n_regionkey = r_regionkey
+ and r_name = 'ASIA'
+ and o_orderdate >= date '1994-01-01'
+ and o_orderdate < date '1994-01-01' + interval '1' year
+ group by
+ n_name
+ order by
+ revenue desc;
+ """
+
+ qt_select """
+
+ select /*+SET_VAR(exec_mem_limit=8589934592, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=true) */
+ n_name,
+ sum(l_extendedprice * (1 - l_discount)) as revenue
+ from
+ customer,
+ orders,
+ lineitem,
+ supplier,
+ nation,
+ region
+ where
+ c_custkey = o_custkey
+ and l_orderkey = o_orderkey
+ and l_suppkey = s_suppkey
+ and c_nationkey = s_nationkey
+ and s_nationkey = n_nationkey
+ and n_regionkey = r_regionkey
+ and r_name = 'ASIA'
+ and o_orderdate >= date '1994-01-01'
+ and o_orderdate < date '1994-01-01' + interval '1' year
+ group by
+ n_name
+ order by
+ revenue desc;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q6.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q6.groovy
new file mode 100644
index 0000000000..1eaef71f85
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q6.groovy
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q6_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ sum(l_extendedprice * l_discount) as revenue
+ from
+ lineitem
+ where
+ l_shipdate >= date '1994-01-01'
+ and l_shipdate < date '1994-01-01' + interval '1' year
+ and l_discount between .06 - 0.01 and .06 + 0.01
+ and l_quantity < 24;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=1, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=true) */
+ sum(l_extendedprice * l_discount) as revenue
+ from
+ lineitem
+ where
+ l_shipdate >= date '1994-01-01'
+ and l_shipdate < date '1994-01-01' + interval '1' year
+ and l_discount between .06 - 0.01 and .06 + 0.01
+ and l_quantity < 24;
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q7.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q7.groovy
new file mode 100644
index 0000000000..fd46be98ea
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q7.groovy
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q7_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+ select
+ supp_nation,
+ cust_nation,
+ l_year,
+ sum(volume) as revenue
+ from
+ (
+ select
+ n1.n_name as supp_nation,
+ n2.n_name as cust_nation,
+ extract(year from l_shipdate) as l_year,
+ l_extendedprice * (1 - l_discount) as volume
+ from
+ supplier,
+ lineitem,
+ orders,
+ customer,
+ nation n1,
+ nation n2
+ where
+ s_suppkey = l_suppkey
+ and o_orderkey = l_orderkey
+ and c_custkey = o_custkey
+ and s_nationkey = n1.n_nationkey
+ and c_nationkey = n2.n_nationkey
+ and (
+ (n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY')
+ or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')
+ )
+ and l_shipdate between date '1995-01-01' and date '1996-12-31'
+ ) as shipping
+ group by
+ supp_nation,
+ cust_nation,
+ l_year
+ order by
+ supp_nation,
+ cust_nation,
+ l_year;
+ """
+
+ qt_select """
+ select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=true) */
+ supp_nation,
+ cust_nation,
+ l_year,
+ sum(volume) as revenue
+ from
+ (
+ select
+ n1.n_name as supp_nation,
+ n2.n_name as cust_nation,
+ extract(year from l_shipdate) as l_year,
+ l_extendedprice * (1 - l_discount) as volume
+ from
+ supplier,
+ lineitem,
+ orders,
+ customer,
+ nation n1,
+ nation n2
+ where
+ s_suppkey = l_suppkey
+ and o_orderkey = l_orderkey
+ and c_custkey = o_custkey
+ and s_nationkey = n1.n_nationkey
+ and c_nationkey = n2.n_nationkey
+ and (
+ (n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY')
+ or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')
+ )
+ and l_shipdate between date '1995-01-01' and date '1996-12-31'
+ ) as shipping
+ group by
+ supp_nation,
+ cust_nation,
+ l_year
+ order by
+ supp_nation,
+ cust_nation,
+ l_year;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q8.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q8.groovy
new file mode 100644
index 0000000000..86caa36cf1
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q8.groovy
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q8_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+
+ qt_select """
+select
+ o_year,
+ sum(case
+ when nation = 'BRAZIL' then volume
+ else 0
+ end) / sum(volume) as mkt_share
+from
+ (
+ select
+ extract(year from o_orderdate) as o_year,
+ l_extendedprice * (1 - l_discount) as volume,
+ n2.n_name as nation
+ from
+ part,
+ supplier,
+ lineitem,
+ orders,
+ customer,
+ nation n1,
+ nation n2,
+ region
+ where
+ p_partkey = l_partkey
+ and s_suppkey = l_suppkey
+ and l_orderkey = o_orderkey
+ and o_custkey = c_custkey
+ and c_nationkey = n1.n_nationkey
+ and n1.n_regionkey = r_regionkey
+ and r_name = 'AMERICA'
+ and s_nationkey = n2.n_nationkey
+ and o_orderdate between date '1995-01-01' and date '1996-12-31'
+ and p_type = 'ECONOMY ANODIZED STEEL'
+ ) as all_nations
+group by
+ o_year
+order by
+ o_year;
+ """
+
+ qt_select """
+
+select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */
+ o_year,
+ sum(case
+ when nation = 'BRAZIL' then volume
+ else 0
+ end) / sum(volume) as mkt_share
+from
+ (
+ select
+ extract(year from o_orderdate) as o_year,
+ l_extendedprice * (1 - l_discount) as volume,
+ n2.n_name as nation
+ from
+ lineitem,
+ orders,
+ customer,
+ supplier,
+ part,
+ nation n1,
+ nation n2,
+ region
+ where
+ p_partkey = l_partkey
+ and s_suppkey = l_suppkey
+ and l_orderkey = o_orderkey
+ and o_custkey = c_custkey
+ and c_nationkey = n1.n_nationkey
+ and n1.n_regionkey = r_regionkey
+ and r_name = 'AMERICA'
+ and s_nationkey = n2.n_nationkey
+ and o_orderdate between date '1995-01-01' and date '1996-12-31'
+ and p_type = 'ECONOMY ANODIZED STEEL'
+ ) as all_nations
+group by
+ o_year
+order by
+ o_year;
+
+ """
+
+}
diff --git a/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q9.groovy b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q9.groovy
new file mode 100644
index 0000000000..89055edec8
--- /dev/null
+++ b/regression-test/suites/tpch_sf1_p1/tpch_sf1/nereids/q9.groovy
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+suite("tpch_sf1_q9_nereids") {
+ String realDb = context.config.getDbNameByFile(context.file)
+ // get parent directory's group
+ realDb = realDb.substring(0, realDb.lastIndexOf("_"))
+
+ sql "use ${realDb}"
+
+ sql 'set enable_nereids_planner=true'
+ sql 'set enable_fallback_to_original_planner=false'
+ sql 'set exec_mem_limit=17179869184'
+
+ qt_select """
+ select
+ nation,
+ o_year,
+ sum(amount) as sum_profit
+ from
+ (
+ select
+ n_name as nation,
+ extract(year from o_orderdate) as o_year,
+ l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
+ from
+ part,
+ supplier,
+ lineitem,
+ partsupp,
+ orders,
+ nation
+ where
+ s_suppkey = l_suppkey
+ and ps_suppkey = l_suppkey
+ and ps_partkey = l_partkey
+ and p_partkey = l_partkey
+ and o_orderkey = l_orderkey
+ and s_nationkey = n_nationkey
+ and p_name like '%green%'
+ ) as profit
+ group by
+ nation,
+ o_year
+ order by
+ nation,
+ o_year desc;
+ """
+
+ qt_select """
+ select/*+SET_VAR(exec_mem_limit=17179869184, parallel_fragment_exec_instance_num=4, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=true, enable_remove_no_conjuncts_runtime_filter_policy=true, runtime_filter_wait_time_ms=10000) */
+ nation,
+ o_year,
+ sum(amount) as sum_profit
+ from
+ (
+ select
+ n_name as nation,
+ extract(year from o_orderdate) as o_year,
+ l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
+ from
+ lineitem join orders on o_orderkey = l_orderkey
+ join part on p_partkey = l_partkey
+ join partsupp on ps_partkey = l_partkey
+ join supplier on s_suppkey = l_suppkey
+ join nation on s_nationkey = n_nationkey
+ where
+ ps_suppkey = l_suppkey and
+ p_name like '%green%'
+ ) as profit
+ group by
+ nation,
+ o_year
+ order by
+ nation,
+ o_year desc;
+ """
+
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 13/39: [fix](statistics) ColumnStatistics was changed unexpectedly when show stats (#14068)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bbc60f4e99baaa01bdfda8547c766086638df099
Author: Kikyou1997 <33...@users.noreply.github.com>
AuthorDate: Tue Nov 8 20:26:37 2022 +0800
[fix](statistics) ColumnStatistics was changed unexpectedly when show stats (#14068)
The logic of show stats would change the internal collected ColumnStat unexpectedly which would cause inaccurate cost and inefficient plan
---
.../main/java/org/apache/doris/common/Config.java | 4 +-
.../apache/doris/statistics/OlapTableStats.java | 46 ----------------------
.../org/apache/doris/statistics/TableStats.java | 2 +-
.../org/apache/doris/statistics/TabletStats.java | 42 --------------------
4 files changed, 3 insertions(+), 91 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/Config.java b/fe/fe-core/src/main/java/org/apache/doris/common/Config.java
index a1208ff673..24534a2caf 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/Config.java
@@ -1766,10 +1766,10 @@ public class Config extends ConfigBase {
public static int be_exec_version = max_be_exec_version;
@ConfField(mutable = false)
- public static int statistic_job_scheduler_execution_interval_ms = 60 * 1000;
+ public static int statistic_job_scheduler_execution_interval_ms = 1000;
@ConfField(mutable = false)
- public static int statistic_task_scheduler_execution_interval_ms = 60 * 1000;
+ public static int statistic_task_scheduler_execution_interval_ms = 1000;
/*
* mtmv scheduler framework is still under dev, remove this config when it is graduate.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapTableStats.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapTableStats.java
deleted file mode 100644
index 4ab0d20702..0000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapTableStats.java
+++ /dev/null
@@ -1,46 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.statistics;
-
-import com.google.common.collect.Maps;
-
-import java.util.Map;
-
-/**
- * There are the statistics of OlapTable.
- * The @OlapTableStats are mainly used to provide input for the Optimizer's cost model.
- *
- * There are three kinds of statistics of OlapTable.
- * @rowCount: The row count of OlapTable. There are two ways to obtain value:
- * 1. The sum row count of @TabletStats which maybe an inaccurate value.
- * 2. count(*) of OlapTable which is an accurate value.
- * @dataSize: The data size of OlapTable. This is an inaccurate value,
- * which is obtained by summing the @dataSize of @TabletStats.
- * @idToTabletStats: <@Long tabletId, @TabletStats tabletStats>
- * Each tablet in the OlapTable will have corresponding @TabletStats.
- * Those @TabletStats are recorded in @idToTabletStats form of MAP.
- * This facilitates the optimizer to quickly find the corresponding
- * @TabletStats based on the tablet id.
- * At the same time, both @rowCount and @dataSize can also be obtained
- * from the sum of all @TabletStats.
- *
- */
-public class OlapTableStats extends TableStats {
-
- private Map<Long, TabletStats> idToTabletStats = Maps.newHashMap();
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java
index 93aeb49431..ad840dd70b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStats.java
@@ -225,7 +225,7 @@ public class TableStats {
for (PartitionStats partitionStats : nameToPartitionStats.values()) {
partitionStats.getNameToColumnStats().forEach((colName, columnStats) -> {
if (!aggColumnStats.containsKey(colName)) {
- aggColumnStats.put(colName, columnStats);
+ aggColumnStats.put(colName, columnStats.copy());
} else {
ColumnStat tblColStats = aggColumnStats.get(colName);
mergePartitionColumnStats(tblColStats, columnStats);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/TabletStats.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/TabletStats.java
deleted file mode 100644
index 756b82d2de..0000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/TabletStats.java
+++ /dev/null
@@ -1,42 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.statistics;
-
-/**
- * There are the statistics of one tablet.
- * The tablet stats are mainly used to provide input for the Optimizer's cost model.
- *
- * The description of tablet stats are following:
- * 1. @rowCount: The row count of tablet.
- * 2. @dataSize: The data size of tablet.
- *
- * @rowCount: The row count of tablet. There are two ways to update:
- * 1. The rowCount from tablet meta. The value obtained by this update method
- * may be an inaccurate value.
- * 2. The result of count(*) query from one tablet. The value obtained by this update method
- * is accurate.
- * @dataSize: The data size of tablet. This is a inaccurate value of one tablet.
- *
- * The granularity of the statistics is one tablet.
- * For example:
- * "@rowCount = 10" means that the row count is 1000 in one tablet.
- */
-public class TabletStats {
- private long rowCount;
- private long dataSize;
-}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 34/39: [performance-wip] (vectorization) Opt HashJoin Performance (#12390)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1eabf6975a780b2f3e2b29a88be549e3ad2fb398
Author: WenYao <72...@qq.com>
AuthorDate: Wed Nov 9 14:07:49 2022 +0800
[performance-wip] (vectorization) Opt HashJoin Performance (#12390)
---
be/src/vec/common/hash_table/join_hash_map.h | 133 +++++
be/src/vec/common/hash_table/join_hash_table.h | 733 +++++++++++++++++++++++++
2 files changed, 866 insertions(+)
diff --git a/be/src/vec/common/hash_table/join_hash_map.h b/be/src/vec/common/hash_table/join_hash_map.h
new file mode 100644
index 0000000000..089e968766
--- /dev/null
+++ b/be/src/vec/common/hash_table/join_hash_map.h
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Common/HashTable/HashMap.h
+// and modified by Doris
+
+#pragma once
+
+#include "vec/common/hash_table/hash_map.h"
+#include "vec/common/hash_table/join_hash_table.h"
+
+/** NOTE JoinHashMap could only be used for memmoveable (position independent) types.
+ * Example: std::string is not position independent in libstdc++ with C++11 ABI or in libc++.
+ * Also, key in hash table must be of type, that zero bytes is compared equals to zero key.
+ */
+
+template <typename Key, typename Cell, typename Hash = DefaultHash<Key>,
+ typename Grower = HashTableGrower<>, typename Allocator = HashTableAllocator>
+class JoinHashMapTable : public JoinHashTable<Key, Cell, Hash, Grower, Allocator> {
+public:
+ using Self = JoinHashMapTable;
+ using Base = JoinHashTable<Key, Cell, Hash, Grower, Allocator>;
+
+ using key_type = Key;
+ using value_type = typename Cell::value_type;
+ using mapped_type = typename Cell::Mapped;
+
+ using LookupResult = typename Base::LookupResult;
+
+ using JoinHashTable<Key, Cell, Hash, Grower, Allocator>::JoinHashTable;
+
+ /// Merge every cell's value of current map into the destination map via emplace.
+ /// Func should have signature void(Mapped & dst, Mapped & src, bool emplaced).
+ /// Each filled cell in current map will invoke func once. If that map doesn't
+ /// have a key equals to the given cell, a new cell gets emplaced into that map,
+ /// and func is invoked with the third argument emplaced set to true. Otherwise
+ /// emplaced is set to false.
+ template <typename Func>
+ void ALWAYS_INLINE merge_to_via_emplace(Self& that, Func&& func) {
+ for (auto it = this->begin(), end = this->end(); it != end; ++it) {
+ typename Self::LookupResult res_it;
+ bool inserted;
+ that.emplace(it->get_first(), res_it, inserted, it.get_hash());
+ func(*lookup_result_get_mapped(res_it), it->get_second(), inserted);
+ }
+ }
+
+ /// Merge every cell's value of current map into the destination map via find.
+ /// Func should have signature void(Mapped & dst, Mapped & src, bool exist).
+ /// Each filled cell in current map will invoke func once. If that map doesn't
+ /// have a key equals to the given cell, func is invoked with the third argument
+ /// exist set to false. Otherwise exist is set to true.
+ template <typename Func>
+ void ALWAYS_INLINE merge_to_via_find(Self& that, Func&& func) {
+ for (auto it = this->begin(), end = this->end(); it != end; ++it) {
+ auto res_it = that.find(it->get_first(), it.get_hash());
+ if (!res_it)
+ func(it->get_second(), it->get_second(), false);
+ else
+ func(*lookup_result_get_mapped(res_it), it->get_second(), true);
+ }
+ }
+
+ /// Call func(const Key &, Mapped &) for each hash map element.
+ template <typename Func>
+ void for_each_value(Func&& func) {
+ for (auto& v : *this) func(v.get_first(), v.get_second());
+ }
+
+ /// Call func(Mapped &) for each hash map element.
+ template <typename Func>
+ void for_each_mapped(Func&& func) {
+ for (auto& v : *this) func(v.get_second());
+ }
+
+ size_t get_size() {
+ size_t count = 0;
+ for (auto& v : *this) {
+ count += v.get_second().get_row_count();
+ }
+ return count;
+ }
+
+ mapped_type& ALWAYS_INLINE operator[](Key x) {
+ typename JoinHashMapTable::LookupResult it;
+ bool inserted;
+ this->emplace(x, it, inserted);
+
+ /** It may seem that initialization is not necessary for POD-types (or __has_trivial_constructor),
+ * since the hash table memory is initially initialized with zeros.
+ * But, in fact, an empty cell may not be initialized with zeros in the following cases:
+ * - ZeroValueStorage (it only zeros the key);
+ * - after resizing and moving a part of the cells to the new half of the hash table, the old cells also have only the key to zero.
+ *
+ * On performance, there is almost always no difference, due to the fact that it->second is usually assigned immediately
+ * after calling `operator[]`, and since `operator[]` is inlined, the compiler removes unnecessary initialization.
+ *
+ * Sometimes due to initialization, the performance even grows. This occurs in code like `++map[key]`.
+ * When we do the initialization, for new cells, it's enough to make `store 1` right away.
+ * And if we did not initialize, then even though there was zero in the cell,
+ * the compiler can not guess about this, and generates the `load`, `increment`, `store` code.
+ */
+ if (inserted) new (lookup_result_get_mapped(it)) mapped_type();
+
+ return *lookup_result_get_mapped(it);
+ }
+
+ char* get_null_key_data() { return nullptr; }
+ bool has_null_key_data() const { return false; }
+};
+
+template <typename Key, typename Mapped, typename Hash = DefaultHash<Key>,
+ typename Grower = JoinHashTableGrower<>, typename Allocator = HashTableAllocator>
+using JoinHashMap = JoinHashMapTable<Key, HashMapCell<Key, Mapped, Hash>, Hash, Grower, Allocator>;
+
+template <typename Key, typename Mapped, typename Hash = DefaultHash<Key>,
+ typename Grower = JoinHashTableGrower<>, typename Allocator = HashTableAllocator>
+using JoinHashMapWithSavedHash =
+ JoinHashMapTable<Key, HashMapCellWithSavedHash<Key, Mapped, Hash>, Hash, Grower, Allocator>;
\ No newline at end of file
diff --git a/be/src/vec/common/hash_table/join_hash_table.h b/be/src/vec/common/hash_table/join_hash_table.h
new file mode 100644
index 0000000000..5e51eeb62e
--- /dev/null
+++ b/be/src/vec/common/hash_table/join_hash_table.h
@@ -0,0 +1,733 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/common/allocator.h"
+#include "vec/common/hash_table/hash_table.h"
+
+/** NOTE JoinHashTable could only be used for memmoveable (position independent) types.
+ * Example: std::string is not position independent in libstdc++ with C++11 ABI or in libc++.
+ * Also, key in hash table must be of type, that zero bytes is compared equals to zero key.
+ */
+
+/** Determines the size of the join hash table, and when and how much it should be resized.
+ */
+template <size_t initial_size_degree = 10>
+struct JoinHashTableGrower {
+ /// The state of this structure is enough to get the buffer size of the join hash table.
+ doris::vectorized::UInt8 size_degree = initial_size_degree;
+ doris::vectorized::Int64 double_grow_degree = 31; // 2GB
+
+ size_t bucket_size() const { return 1ULL << (size_degree - 1); }
+
+ /// The size of the join hash table in the cells.
+ size_t buf_size() const { return 1ULL << size_degree; }
+
+ size_t max_fill() const { return buf_size(); }
+
+ size_t mask() const { return bucket_size() - 1; }
+
+ /// From the hash value, get the bucket id (first index) in the join hash table.
+ size_t place(size_t x) const { return x & mask(); }
+
+ /// Whether the join hash table is full. You need to increase the size of the hash table, or remove something unnecessary from it.
+ bool overflow(size_t elems) const { return elems >= max_fill(); }
+
+ /// Increase the size of the join hash table.
+ void increase_size() { size_degree += size_degree >= 23 ? 1 : 2; }
+
+ /// Set the buffer size by the number of elements in the join hash table. Used when deserializing a join hash table.
+ void set(size_t num_elems) {
+#ifndef STRICT_MEMORY_USE
+ size_t fill_capacity = static_cast<size_t>(log2(num_elems - 1)) + 2;
+#else
+ size_t fill_capacity = static_cast<size_t>(log2(num_elems - 1)) + 1;
+ fill_capacity =
+ fill_capacity < double_grow_degree
+ ? fill_capacity + 1
+ : (num_elems < (1ULL << fill_capacity) - (1ULL << (fill_capacity - 2))
+ ? fill_capacity
+ : fill_capacity + 1);
+#endif
+ size_degree = num_elems <= 1 ? initial_size_degree
+ : (initial_size_degree > fill_capacity ? initial_size_degree
+ : fill_capacity);
+ }
+
+ void set_buf_size(size_t buf_size_) {
+ size_degree = static_cast<size_t>(log2(buf_size_ - 1) + 1);
+ }
+};
+
+/** Determines the size of the join hash table, and when and how much it should be resized.
+ * This structure is aligned to cache line boundary and also occupies it all.
+ * Precalculates some values to speed up lookups and insertion into the JoinHashTable (and thus has bigger memory footprint than JoinHashTableGrower).
+ */
+template <size_t initial_size_degree = 8>
+class alignas(64) JoinHashTableGrowerWithPrecalculation {
+ /// The state of this structure is enough to get the buffer size of the join hash table.
+
+ doris::vectorized::UInt8 size_degree_ = initial_size_degree;
+ size_t precalculated_mask = (1ULL << (initial_size_degree - 1)) - 1;
+ size_t precalculated_max_fill = 1ULL << initial_size_degree;
+
+public:
+ doris::vectorized::UInt8 size_degree() const { return size_degree_; }
+
+ void increase_size_degree(doris::vectorized::UInt8 delta) {
+ size_degree_ += delta;
+ precalculated_mask = (1ULL << (size_degree_ - 1)) - 1;
+ precalculated_max_fill = 1ULL << size_degree_;
+ }
+
+ static constexpr auto initial_count = 1ULL << initial_size_degree;
+
+ /// If collision resolution chains are contiguous, we can implement erase operation by moving the elements.
+ static constexpr auto performs_linear_probing_with_single_step = true;
+
+ size_t bucket_size() const { return 1ULL << (size_degree_ - 1); }
+
+ /// The size of the join hash table in the cells.
+ size_t buf_size() const { return 1ULL << size_degree_; }
+
+ /// From the hash value, get the cell number in the join hash table.
+ size_t place(size_t x) const { return x & precalculated_mask; }
+
+ /// Whether the join hash table is full. You need to increase the size of the hash table, or remove something unnecessary from it.
+ bool overflow(size_t elems) const { return elems >= precalculated_max_fill; }
+
+ /// Increase the size of the join hash table.
+ void increase_size() { increase_size_degree(size_degree_ >= 23 ? 1 : 2); }
+
+ /// Set the buffer size by the number of elements in the join hash table. Used when deserializing a join hash table.
+ void set(size_t num_elems) {
+ size_degree_ =
+ num_elems <= 1
+ ? initial_size_degree
+ : ((initial_size_degree > static_cast<size_t>(log2(num_elems - 1)) + 2)
+ ? initial_size_degree
+ : (static_cast<size_t>(log2(num_elems - 1)) + 2));
+ increase_size_degree(0);
+ }
+
+ void set_buf_size(size_t buf_size_) {
+ size_degree_ = static_cast<size_t>(log2(buf_size_ - 1) + 1);
+ increase_size_degree(0);
+ }
+};
+
+static_assert(sizeof(JoinHashTableGrowerWithPrecalculation<>) == 64);
+
+/** When used as a Grower, it turns a hash table into something like a lookup table.
+ * It remains non-optimal - the cells store the keys.
+ * Also, the compiler can not completely remove the code of passing through the collision resolution chain, although it is not needed.
+ * TODO Make a proper lookup table.
+ */
+template <size_t key_bits>
+struct JoinHashTableFixedGrower {
+ size_t bucket_size() const { return 1ULL << (key_bits - 1); }
+ size_t buf_size() const { return 1ULL << key_bits; }
+ size_t place(size_t x) const { return x & (bucket_size() - 1); }
+ bool overflow(size_t /*elems*/) const { return false; }
+
+ void increase_size() { __builtin_unreachable(); }
+ void set(size_t /*num_elems*/) {}
+ void set_buf_size(size_t /*buf_size_*/) {}
+};
+
+template <typename Key, typename Cell, typename Hash, typename Grower, typename Allocator>
+class JoinHashTable : private boost::noncopyable,
+ protected Hash,
+ protected Allocator,
+ protected Cell::State,
+ protected ZeroValueStorage<Cell::need_zero_value_storage,
+ Cell> /// empty base optimization
+{
+protected:
+ friend class const_iterator;
+ friend class iterator;
+ friend class Reader;
+
+ template <typename, typename, typename, typename, typename, typename, size_t>
+ friend class TwoLevelHashTable;
+
+ template <typename SubMaps>
+ friend class StringHashTable;
+
+ using HashValue = size_t;
+ using Self = JoinHashTable;
+ using cell_type = Cell;
+
+ size_t m_size = 0; /// Amount of elements
+ size_t m_no_zero_size = 0; /// Amount of elements except the element with zero key.
+ Cell* buf; /// A piece of memory for all elements except the element with zero key.
+
+ // bucket-chained hash table
+ // "first" is the buckets of the hash map, and it holds the index of the first key value saved in each bucket,
+ // while other keys can be found by following the indices saved in
+ // "next". "next[0]" represents the end of the list of keys in a bucket.
+ // https://dare.uva.nl/search?identifier=5ccbb60a-38b8-4eeb-858a-e7735dd37487
+ size_t* first;
+ size_t* next;
+
+ Grower grower;
+ int64_t _resize_timer_ns;
+
+ //factor that will trigger growing the hash table on insert.
+ static constexpr float MAX_BUCKET_OCCUPANCY_FRACTION = 1.0f;
+
+#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
+ mutable size_t collisions = 0;
+#endif
+
+ /// Find a cell with the same key or an empty cell, starting from the specified position and further along the collision resolution chain.
+ size_t ALWAYS_INLINE find_cell(const Key& x, size_t hash_value, size_t place_value) const {
+ while (place_value && !buf[place_value - 1].is_zero(*this) &&
+ !buf[place_value - 1].key_equals(x, hash_value, *this)) {
+ place_value = next[place_value];
+#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
+ ++collisions;
+#endif
+ }
+
+ return place_value;
+ }
+
+ std::pair<bool, size_t> ALWAYS_INLINE find_cell_opt(const Key& x, size_t hash_value,
+ size_t place_value) const {
+ bool is_zero = false;
+ do {
+ if (!place_value) return {true, place_value};
+ is_zero = buf[place_value - 1].is_zero(*this); ///
+ if (is_zero || buf[place_value - 1].key_equals(x, hash_value, *this)) break;
+ place_value = next[place_value];
+ } while (true);
+
+ return {is_zero, place_value};
+ }
+
+ /// Find an empty cell, starting with the specified position and further along the collision resolution chain.
+ size_t ALWAYS_INLINE find_empty_cell(size_t place_value) const {
+ while (place_value && !buf[place_value - 1].is_zero(*this)) {
+ place_value = next[place_value];
+#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
+ ++collisions;
+#endif
+ }
+
+ return place_value;
+ }
+
+ void alloc(const Grower& new_grower) {
+ buf = reinterpret_cast<Cell*>(Allocator::alloc(new_grower.buf_size() * sizeof(Cell)));
+ first = reinterpret_cast<size_t*>(
+ Allocator::alloc(new_grower.bucket_size() * sizeof(size_t)));
+ memset(first, 0, new_grower.bucket_size() * sizeof(size_t));
+ next = reinterpret_cast<size_t*>(
+ Allocator::alloc((new_grower.buf_size() + 1) * sizeof(size_t)));
+ memset(next, 0, (new_grower.buf_size() + 1) * sizeof(size_t));
+ grower = new_grower;
+ }
+
+ void free() {
+ if (buf) {
+ Allocator::free(buf, get_buffer_size_in_bytes());
+ buf = nullptr;
+ }
+ if (first) {
+ Allocator::free(first, grower.bucket_size() * sizeof(size_t));
+ first = nullptr;
+ }
+ if (next) {
+ Allocator::free(next, (grower.buf_size() + 1) * sizeof(size_t));
+ next = nullptr;
+ }
+ }
+
+ /// Increase the size of the buffer.
+ void resize(size_t for_num_elems = 0, size_t for_buf_size = 0) {
+ SCOPED_RAW_TIMER(&_resize_timer_ns);
+#ifdef DBMS_HASH_MAP_DEBUG_RESIZES
+ Stopwatch watch;
+#endif
+
+ size_t old_size = grower.buf_size();
+
+ /** In case of exception for the object to remain in the correct state,
+ * changing the variable `grower` (which determines the buffer size of the hash table)
+ * is postponed for a moment after a real buffer change.
+ * The temporary variable `new_grower` is used to determine the new size.
+ */
+ Grower new_grower = grower;
+ if (for_num_elems) {
+ new_grower.set(for_num_elems);
+ if (new_grower.buf_size() <= old_size) return;
+ } else if (for_buf_size) {
+ new_grower.set_buf_size(for_buf_size);
+ if (new_grower.buf_size() <= old_size) return;
+ } else
+ new_grower.increase_size();
+
+ /// Expand the space.
+ buf = reinterpret_cast<Cell*>(Allocator::realloc(buf, get_buffer_size_in_bytes(),
+ new_grower.buf_size() * sizeof(Cell)));
+ first = reinterpret_cast<size_t*>(Allocator::realloc(
+ first, get_bucket_size_in_bytes(), new_grower.bucket_size() * sizeof(size_t)));
+ memset(first, 0, new_grower.bucket_size() * sizeof(size_t));
+ next = reinterpret_cast<size_t*>(Allocator::realloc(
+ next, get_buffer_size_in_bytes(), (new_grower.buf_size() + 1) * sizeof(size_t)));
+ memset(next, 0, (new_grower.buf_size() + 1) * sizeof(size_t));
+ grower = new_grower;
+
+ /** Now some items may need to be moved to a new location.
+ * The element can stay in place, or move to a new location "on the right",
+ * or move to the left of the collision resolution chain, because the elements to the left of it have been moved to the new "right" location.
+ */
+ size_t i = 0;
+ for (; i < m_no_zero_size; ++i)
+ if (!buf[i].is_zero(*this)) reinsert(i + 1, buf[i], buf[i].get_hash(*this));
+
+#ifdef DBMS_HASH_MAP_DEBUG_RESIZES
+ watch.stop();
+ std::cerr << std::fixed << std::setprecision(3) << "Resize from " << old_size << " to "
+ << grower.buf_size() << " took " << watch.elapsedSeconds() << " sec."
+ << std::endl;
+#endif
+ }
+
+ /** Paste into the new buffer the value that was in the old buffer.
+ * Used when increasing the buffer size.
+ */
+ void reinsert(size_t place_value, Cell& x, size_t hash_value) {
+ size_t bucket_value = grower.place(hash_value);
+ next[place_value] = first[bucket_value];
+ first[bucket_value] = place_value;
+ }
+
+ void destroy_elements() {
+ if (!std::is_trivially_destructible_v<Cell>)
+ for (iterator it = begin(), it_end = end(); it != it_end; ++it) it.ptr->~Cell();
+ }
+
+ template <typename Derived, bool is_const>
+ class iterator_base {
+ using Container = std::conditional_t<is_const, const Self, Self>;
+ using cell_type = std::conditional_t<is_const, const Cell, Cell>;
+
+ Container* container;
+ cell_type* ptr;
+
+ friend class JoinHashTable;
+
+ public:
+ iterator_base() {}
+ iterator_base(Container* container_, cell_type* ptr_) : container(container_), ptr(ptr_) {}
+
+ bool operator==(const iterator_base& rhs) const { return ptr == rhs.ptr; }
+ bool operator!=(const iterator_base& rhs) const { return ptr != rhs.ptr; }
+
+ Derived& operator++() {
+ /// If iterator was pointed to ZeroValueStorage, move it to the beginning of the main buffer.
+ if (UNLIKELY(ptr->is_zero(*container)))
+ ptr = container->buf;
+ else
+ ++ptr;
+
+ /// Skip empty cells in the main buffer.
+ auto buf_end = container->buf + container->m_no_zero_size;
+ while (ptr < buf_end && ptr->is_zero(*container)) ++ptr;
+
+ return static_cast<Derived&>(*this);
+ }
+
+ auto& operator*() const { return *ptr; }
+ auto* operator->() const { return ptr; }
+
+ auto get_ptr() const { return ptr; }
+ size_t get_hash() const { return ptr->get_hash(*container); }
+
+ size_t get_collision_chain_length() const { ////////////// ?????????
+ return 0;
+ }
+
+ operator Cell*() const { return nullptr; }
+ };
+
+public:
+ using key_type = Key;
+ using value_type = typename Cell::value_type;
+
+ // Use lookup_result_get_mapped/Key to work with these values.
+ using LookupResult = Cell*;
+ using ConstLookupResult = const Cell*;
+
+ void reset_resize_timer() { _resize_timer_ns = 0; }
+ int64_t get_resize_timer_value() const { return _resize_timer_ns; }
+
+ size_t hash(const Key& x) const { return Hash::operator()(x); }
+
+ JoinHashTable() {
+ if (Cell::need_zero_value_storage) this->zero_value()->set_zero();
+ alloc(grower);
+ }
+
+ JoinHashTable(size_t reserve_for_num_elements) {
+ if (Cell::need_zero_value_storage) this->zero_value()->set_zero();
+ grower.set(reserve_for_num_elements);
+ alloc(grower);
+ }
+
+ JoinHashTable(JoinHashTable&& rhs) : buf(nullptr) { *this = std::move(rhs); }
+
+ ~JoinHashTable() {
+ destroy_elements();
+ free();
+ }
+
+ JoinHashTable& operator=(JoinHashTable&& rhs) {
+ destroy_elements();
+ free();
+
+ std::swap(buf, rhs.buf);
+ std::swap(m_size, rhs.m_size);
+ std::swap(m_no_zero_size, rhs.m_no_zero_size);
+ std::swap(first, rhs.first);
+ std::swap(next, rhs.next);
+ std::swap(grower, rhs.grower);
+
+ Hash::operator=(std::move(rhs));
+ Allocator::operator=(std::move(rhs));
+ Cell::State::operator=(std::move(rhs));
+ ZeroValueStorage<Cell::need_zero_value_storage, Cell>::operator=(std::move(rhs));
+
+ return *this;
+ }
+
+ class iterator : public iterator_base<iterator, false> {
+ public:
+ using iterator_base<iterator, false>::iterator_base;
+ };
+
+ class const_iterator : public iterator_base<const_iterator, true> {
+ public:
+ using iterator_base<const_iterator, true>::iterator_base;
+ };
+
+ const_iterator begin() const {
+ if (!buf) return end();
+
+ if (this->get_has_zero()) return iterator_to_zero();
+
+ const Cell* ptr = buf;
+ auto buf_end = buf + m_no_zero_size;
+ while (ptr < buf_end && ptr->is_zero(*this)) ++ptr;
+
+ return const_iterator(this, ptr);
+ }
+
+ const_iterator cbegin() const { return begin(); }
+
+ iterator begin() {
+ if (!buf) return end();
+
+ if (this->get_has_zero()) return iterator_to_zero();
+
+ Cell* ptr = buf;
+ auto buf_end = buf + m_no_zero_size;
+ while (ptr < buf_end && ptr->is_zero(*this)) ++ptr;
+
+ return iterator(this, ptr);
+ }
+
+ const_iterator end() const { return const_iterator(this, buf + m_no_zero_size); }
+ const_iterator cend() const { return end(); }
+ iterator end() { return iterator(this, buf + m_no_zero_size); }
+
+protected:
+ const_iterator iterator_to(const Cell* ptr) const { return const_iterator(this, ptr); }
+ iterator iterator_to(Cell* ptr) { return iterator(this, ptr); }
+ const_iterator iterator_to_zero() const { return iterator_to(this->zero_value()); }
+ iterator iterator_to_zero() { return iterator_to(this->zero_value()); }
+
+ /// If the key is zero, insert it into a special place and return true.
+ /// We don't have to persist a zero key, because it's not actually inserted.
+ /// That's why we just take a Key by value, an not a key holder.
+ bool ALWAYS_INLINE emplace_if_zero(Key x, LookupResult& it, bool& inserted, size_t hash_value) {
+ /// If it is claimed that the zero key can not be inserted into the table.
+ if (!Cell::need_zero_value_storage) return false;
+
+ if (Cell::is_zero(x, *this)) {
+ it = this->zero_value();
+
+ if (!this->get_has_zero()) {
+ ++m_size;
+ this->set_get_has_zero();
+ this->zero_value()->set_hash(hash_value);
+ inserted = true;
+ } else
+ inserted = false;
+
+ return true;
+ }
+
+ return false;
+ }
+
+ /// Only for non-zero keys. Find the right place, insert the key there, if it does not already exist. Set iterator to the cell in output parameter.
+ template <typename KeyHolder>
+ void ALWAYS_INLINE emplace_non_zero(KeyHolder&& key_holder, LookupResult& it, bool& inserted,
+ size_t hash_value) {
+ it = &buf[m_no_zero_size];
+
+ if (!buf[m_no_zero_size].is_zero(*this)) {
+ key_holder_discard_key(key_holder);
+ inserted = false;
+ return;
+ }
+
+ key_holder_persist_key(key_holder);
+ const auto& key = key_holder_get_key(key_holder);
+
+ new (&buf[m_no_zero_size]) Cell(key, *this);
+ buf[m_no_zero_size].set_hash(hash_value);
+ size_t bucket_value = grower.place(hash_value);
+ inserted = true;
+ ++m_size;
+ ++m_no_zero_size;
+ next[m_no_zero_size] = first[bucket_value];
+ first[bucket_value] = m_no_zero_size;
+
+ if (UNLIKELY(grower.overflow(m_size))) {
+ try {
+ resize();
+ } catch (...) {
+ /** If we have not resized successfully, then there will be problems.
+ * There remains a key, but uninitialized mapped-value,
+ * which, perhaps, can not even be called a destructor.
+ */
+ first[bucket_value] = next[m_no_zero_size];
+ next[m_no_zero_size] = 0;
+ --m_size;
+ --m_no_zero_size;
+ buf[m_no_zero_size].set_zero();
+ throw;
+ }
+ }
+ }
+
+public:
+ void expanse_for_add_elem(size_t num_elem) {
+ std::cout << "expanse_for_add_elem\n";
+ if (add_elem_size_overflow(num_elem)) {
+ resize(grower.buf_size() + num_elem);
+ }
+ }
+
+ /// Insert a value. In the case of any more complex values, it is better to use the `emplace` function.
+ std::pair<LookupResult, bool> ALWAYS_INLINE insert(const value_type& x) {
+ std::pair<LookupResult, bool> res;
+ size_t hash_value = hash(Cell::get_key(x));
+ if (!emplace_if_zero(Cell::get_key(x), res.first, res.second, hash_value)) {
+ emplace_non_zero(Cell::get_key(x), res.first, res.second, hash_value);
+ }
+
+ if (res.second) insert_set_mapped(lookup_result_get_mapped(res.first), x);
+
+ return res;
+ }
+
+ template <typename KeyHolder>
+ void ALWAYS_INLINE prefetch(KeyHolder& key_holder) {
+ key_holder_get_key(key_holder);
+ __builtin_prefetch(&buf[m_no_zero_size]);
+ }
+
+ /// Reinsert node pointed to by iterator
+ // void ALWAYS_INLINE reinsert(iterator& it, size_t hash_value) {
+ // reinsert(*it.get_ptr(), hash_value);
+ // }
+
+ /** Insert the key.
+ * Return values:
+ * 'it' -- a LookupResult pointing to the corresponding key/mapped pair.
+ * 'inserted' -- whether a new key was inserted.
+ *
+ * You have to make `placement new` of value if you inserted a new key,
+ * since when destroying a hash table, it will call the destructor!
+ *
+ * Example usage:
+ *
+ * Map::iterator it;
+ * bool inserted;
+ * map.emplace(key, it, inserted);
+ * if (inserted)
+ * new(&it->second) Mapped(value);
+ */
+ template <typename KeyHolder>
+ void ALWAYS_INLINE emplace(KeyHolder&& key_holder, LookupResult& it, bool& inserted) {
+ const auto& key = key_holder_get_key(key_holder);
+ emplace(key_holder, it, inserted, hash(key));
+ }
+
+ template <typename KeyHolder>
+ void ALWAYS_INLINE emplace(KeyHolder&& key_holder, LookupResult& it, bool& inserted,
+ size_t hash_value) {
+ const auto& key = key_holder_get_key(key_holder);
+ if (!emplace_if_zero(key, it, inserted, hash_value))
+ emplace_non_zero(key_holder, it, inserted, hash_value);
+ }
+
+ /// Copy the cell from another hash table. It is assumed that the cell is not zero, and also that there was no such key in the table yet.
+ void ALWAYS_INLINE insert_unique_non_zero(const Cell* cell, size_t hash_value) {
+ memcpy(static_cast<void*>(&buf[m_no_zero_size]), cell, sizeof(*cell));
+ size_t bucket_value = grower.place();
+ ++m_size;
+ ++m_no_zero_size;
+ next[m_no_zero_size] = first[bucket_value];
+ first[bucket_value] = m_no_zero_size;
+
+ if (UNLIKELY(grower.overflow(m_size))) resize();
+ }
+
+ LookupResult ALWAYS_INLINE find(Key x) {
+ if (Cell::is_zero(x, *this)) return this->get_has_zero() ? this->zero_value() : nullptr;
+
+ size_t hash_value = hash(x);
+ auto [is_zero, place_value] = find_cell_opt(x, hash_value, first[grower.place(hash_value)]);
+
+ if (!place_value) return nullptr;
+
+ return !is_zero ? &buf[place_value - 1] : nullptr;
+ }
+
+ ConstLookupResult ALWAYS_INLINE find(Key x) const {
+ return const_cast<std::decay_t<decltype(*this)>*>(this)->find(x);
+ }
+
+ LookupResult ALWAYS_INLINE find(Key x, size_t hash_value) {
+ if (Cell::is_zero(x, *this)) return this->get_has_zero() ? this->zero_value() : nullptr;
+
+ size_t place_value = find_cell(x, hash_value, first[grower.place(hash_value)]);
+
+ if (!place_value) return nullptr;
+
+ return !buf[place_value - 1].is_zero(*this) ? &buf[place_value - 1] : nullptr;
+ }
+
+ bool ALWAYS_INLINE has(Key x) const {
+ if (Cell::is_zero(x, *this)) return this->get_has_zero();
+
+ size_t hash_value = hash(x);
+ size_t place_value = find_cell(x, hash_value, first[grower.place(hash_value)]);
+ return !place_value && !buf[place_value - 1].is_zero(*this);
+ }
+
+ bool ALWAYS_INLINE has(Key x, size_t hash_value) const {
+ if (Cell::is_zero(x, *this)) return this->get_has_zero();
+
+ size_t place_value = find_cell(x, hash_value, first[grower.place(hash_value)]);
+ return !place_value && !buf[place_value - 1].is_zero(*this);
+ }
+
+ void write(doris::vectorized::BufferWritable& wb) const {
+ Cell::State::write(wb);
+ doris::vectorized::write_var_uint(m_size, wb);
+
+ if (this->get_has_zero()) this->zero_value()->write(wb);
+
+ for (auto ptr = buf, buf_end = buf + m_no_zero_size; ptr < buf_end; ++ptr)
+ if (!ptr->is_zero(*this)) ptr->write(wb);
+ }
+
+ void read(doris::vectorized::BufferReadable& rb) {
+ Cell::State::read(rb);
+
+ destroy_elements();
+ this->clear_get_has_zero();
+ m_size = 0;
+
+ size_t new_size = 0;
+ doris::vectorized::read_var_uint(new_size, rb);
+
+ free();
+ Grower new_grower = grower;
+ new_grower.set(new_size);
+ alloc(new_grower);
+
+ for (size_t i = 0; i < new_size; ++i) {
+ Cell x;
+ x.read(rb);
+ insert(Cell::get_key(x.get_value()));
+ }
+ }
+
+ size_t size() const { return m_size; }
+
+ size_t no_zero_size() const { return m_no_zero_size; }
+
+ bool empty() const { return 0 == m_size; }
+
+ float get_factor() const { return MAX_BUCKET_OCCUPANCY_FRACTION; }
+
+ bool should_be_shrink(int64_t valid_row) { return valid_row < get_factor() * (size() / 2.0); }
+
+ void init_buf_size(size_t reserve_for_num_elements) {
+ free();
+ grower.set(reserve_for_num_elements);
+ alloc(grower);
+ }
+
+ void delete_zero_key(Key key) {
+ if (this->get_has_zero() && Cell::is_zero(key, *this)) {
+ --m_size;
+ this->clear_get_has_zero();
+ }
+ }
+
+ void clear() {
+ destroy_elements();
+ this->clear_get_has_zero();
+ m_size = 0;
+ m_no_zero_size = 0;
+
+ memset(static_cast<void*>(buf), 0, grower.buf_size() * sizeof(*buf));
+ }
+
+ /// After executing this function, the table can only be destroyed,
+ /// and also you can use the methods `size`, `empty`, `begin`, `end`.
+ void clear_and_shrink() {
+ destroy_elements();
+ this->clear_get_has_zero();
+ m_size = 0;
+ m_no_zero_size = 0;
+ free();
+ }
+
+ size_t get_buffer_size_in_bytes() const { return grower.buf_size() * sizeof(Cell); }
+
+ size_t get_bucket_size_in_bytes() const { return grower.bucket_size() * sizeof(Cell); }
+
+ size_t get_buffer_size_in_cells() const { return grower.buf_size(); }
+
+ bool add_elem_size_overflow(size_t add_size) const {
+ return grower.overflow(add_size + m_size);
+ }
+#ifdef DBMS_HASH_MAP_COUNT_COLLISIONS
+ size_t getCollisions() const { return collisions; }
+#endif
+};
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 37/39: [enhancement](Nereids) use post-order to generate runtime filter in RuntimeFilterGenerator (#13949)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e8a9151f1397dedd84343b8841256c7e41b55dca
Author: mch_ucchi <41...@users.noreply.github.com>
AuthorDate: Wed Nov 9 14:28:49 2022 +0800
[enhancement](Nereids) use post-order to generate runtime filter in RuntimeFilterGenerator (#13949)
change runtime filter generator from pre-order to post-order, it maybe change the quantity of generated runtime filters.
and the ut will be corrected.
---
.../glue/translator/RuntimeFilterTranslator.java | 5 +-
.../processor/post/RuntimeFilterContext.java | 62 ++--------
.../processor/post/RuntimeFilterGenerator.java | 130 ++++++++++-----------
.../trees/plans/physical/RuntimeFilter.java | 44 -------
.../nereids/postprocess/RuntimeFilterTest.java | 7 +-
5 files changed, 78 insertions(+), 170 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
index 1dc7a0a2c1..09dbb4115b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
@@ -60,9 +60,8 @@ public class RuntimeFilterTranslator {
* @param ctx plan translator context
*/
public void translateRuntimeFilterTarget(Slot slot, OlapScanNode node, PlanTranslatorContext ctx) {
- context.setKVInNormalMap(context.getExprIdToOlapScanNodeSlotRef(),
- slot.getExprId(), ctx.findSlotRef(slot.getExprId()));
- context.setKVInNormalMap(context.getScanNodeOfLegacyRuntimeFilterTarget(), slot, node);
+ context.getExprIdToOlapScanNodeSlotRef().put(slot.getExprId(), ctx.findSlotRef(slot.getExprId()));
+ context.getScanNodeOfLegacyRuntimeFilterTarget().put(slot, node);
}
/**
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
index 5565dcf928..03c7be5f17 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
@@ -19,6 +19,7 @@ package org.apache.doris.nereids.processor.post;
import org.apache.doris.analysis.SlotRef;
import org.apache.doris.common.IdGenerator;
+import org.apache.doris.common.Pair;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.Slot;
@@ -32,12 +33,9 @@ import org.apache.doris.qe.SessionVariable;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
-import org.jetbrains.annotations.NotNull;
-import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Map;
@@ -62,8 +60,10 @@ public class RuntimeFilterContext {
private final Map<PhysicalHashJoin, List<RuntimeFilter>> runtimeFilterOnHashJoinNode = Maps.newHashMap();
- // Alias's child to itself.
- private final Map<Slot, NamedExpression> aliasChildToSelf = Maps.newHashMap();
+ // alias -> alias's child, if there's a key that is alias's child, the key-value will change by this way
+ // Alias(A) = B, now B -> A in map, and encounter Alias(B) -> C, the kv will be C -> A.
+ // you can see disjoint set data structure to learn the processing detailed.
+ private final Map<NamedExpression, Pair<RelationId, NamedExpression>> aliasTransferMap = Maps.newHashMap();
private final Map<Slot, OlapScanNode> scanNodeOfLegacyRuntimeFilterTarget = Maps.newHashMap();
@@ -86,36 +86,21 @@ public class RuntimeFilterContext {
return limits;
}
- public void setTargetExprIdToFilters(ExprId id, RuntimeFilter... filters) {
- Preconditions.checkArgument(Arrays.stream(filters)
- .allMatch(filter -> filter.getTargetExpr().getExprId() == id));
- this.targetExprIdToFilter.computeIfAbsent(id, k -> Lists.newArrayList())
- .addAll(Arrays.asList(filters));
+ public void setTargetExprIdToFilter(ExprId id, RuntimeFilter filter) {
+ Preconditions.checkArgument(filter.getTargetExpr().getExprId() == id);
+ this.targetExprIdToFilter.computeIfAbsent(id, k -> Lists.newArrayList()).add(filter);
}
- public List<RuntimeFilter> getFiltersByTargetExprId(ExprId id) {
- return targetExprIdToFilter.get(id);
- }
-
- public void removeFilters(ExprId id) {
- targetExprIdToFilter.remove(id);
- }
-
- public void setTargetsOnScanNode(RelationId id, Slot... slots) {
- this.targetOnOlapScanNodeMap.computeIfAbsent(id, k -> Lists.newArrayList())
- .addAll(Arrays.asList(slots));
- }
-
- public <K, V> void setKVInNormalMap(@NotNull Map<K, V> map, K key, V value) {
- map.put(key, value);
+ public void setTargetsOnScanNode(RelationId id, Slot slot) {
+ this.targetOnOlapScanNodeMap.computeIfAbsent(id, k -> Lists.newArrayList()).add(slot);
}
public Map<ExprId, SlotRef> getExprIdToOlapScanNodeSlotRef() {
return exprIdToOlapScanNodeSlotRef;
}
- public Map<Slot, NamedExpression> getAliasChildToSelf() {
- return aliasChildToSelf;
+ public Map<NamedExpression, Pair<RelationId, NamedExpression>> getAliasTransferMap() {
+ return aliasTransferMap;
}
public Map<Slot, OlapScanNode> getScanNodeOfLegacyRuntimeFilterTarget() {
@@ -143,14 +128,6 @@ public class RuntimeFilterContext {
return legacyFilters;
}
- public void setLegacyFilter(org.apache.doris.planner.RuntimeFilter filter) {
- this.legacyFilters.add(filter);
- }
-
- public <K, V> boolean checkExistKey(@NotNull Map<K, V> map, K key) {
- return map.containsKey(key);
- }
-
/**
* get nereids runtime filters
* @return nereids runtime filters
@@ -166,21 +143,6 @@ public class RuntimeFilterContext {
return filters;
}
- /**
- * get the slot list of the same olap scan node of the input slot.
- * @param slot slot
- * @return slot list
- */
- public List<NamedExpression> getSlotListOfTheSameSlotAtOlapScanNode(Slot slot) {
- ImmutableList.Builder<NamedExpression> builder = ImmutableList.builder();
- NamedExpression expr = slot;
- do {
- builder.add(expr);
- expr = aliasChildToSelf.get(expr.toSlot());
- } while (expr != null);
- return builder.build();
- }
-
public void setTargetNullCount() {
targetNullCount++;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
index cc8531bafc..51c84f2dd6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
@@ -18,17 +18,23 @@
package org.apache.doris.nereids.processor.post;
import org.apache.doris.common.IdGenerator;
+import org.apache.doris.common.Pair;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.RelationId;
import org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin;
import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
import org.apache.doris.nereids.trees.plans.physical.RuntimeFilter;
+import org.apache.doris.nereids.util.JoinUtils;
import org.apache.doris.planner.RuntimeFilterId;
import org.apache.doris.thrift.TRuntimeFilterType;
@@ -36,7 +42,8 @@ import com.google.common.collect.ImmutableSet;
import java.util.Arrays;
import java.util.List;
-import java.util.Objects;
+import java.util.Map;
+import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Collectors;
@@ -44,15 +51,12 @@ import java.util.stream.Collectors;
* generate runtime filter
*/
public class RuntimeFilterGenerator extends PlanPostProcessor {
-
- private final IdGenerator<RuntimeFilterId> generator = RuntimeFilterId.createGenerator();
-
- private final ImmutableSet<JoinType> deniedJoinType = ImmutableSet.of(
+ private static final ImmutableSet<JoinType> deniedJoinType = ImmutableSet.of(
JoinType.LEFT_ANTI_JOIN,
- JoinType.RIGHT_ANTI_JOIN,
JoinType.FULL_OUTER_JOIN,
JoinType.LEFT_OUTER_JOIN
);
+ private final IdGenerator<RuntimeFilterId> generator = RuntimeFilterId.createGenerator();
/**
* the runtime filter generator run at the phase of post process and plan translation of nereids planner.
@@ -72,94 +76,80 @@ public class RuntimeFilterGenerator extends PlanPostProcessor {
public PhysicalPlan visitPhysicalHashJoin(PhysicalHashJoin<? extends Plan, ? extends Plan> join,
CascadesContext context) {
RuntimeFilterContext ctx = context.getRuntimeFilterContext();
+ Map<NamedExpression, Pair<RelationId, NamedExpression>> aliasTransferMap = ctx.getAliasTransferMap();
+ join.right().accept(this, context);
+ join.left().accept(this, context);
if (deniedJoinType.contains(join.getJoinType())) {
- /* TODO: translate left outer join to inner join if there are inner join ancestors
- * if it has encountered inner join, like
- * a=b
- * / \
- * / \
- * / \
- * / \
- * left join-->a=c b
- * / \
- * / \
- * / \
- * / \
- * a c
- * runtime filter whose src expr is b can take effect on c.
- * but now checking the inner join is unsupported. we may support it at later version.
- */
- join.getOutput().forEach(slot -> ctx.removeFilters(slot.getExprId()));
+ // copy to avoid bug when next call of getOutputSet()
+ Set<Slot> slots = join.getOutputSet();
+ slots.forEach(aliasTransferMap::remove);
} else {
- List<TRuntimeFilterType> legalTypes = Arrays.stream(TRuntimeFilterType.values()).filter(type ->
- (type.getValue() & ctx.getSessionVariable().getRuntimeFilterType()) > 0)
+ List<TRuntimeFilterType> legalTypes = Arrays.stream(TRuntimeFilterType.values())
+ .filter(type -> (type.getValue() & ctx.getSessionVariable().getRuntimeFilterType()) > 0)
.collect(Collectors.toList());
AtomicInteger cnt = new AtomicInteger();
join.getHashJoinConjuncts().stream()
.map(EqualTo.class::cast)
+ // TODO: some complex situation cannot be handled now, see testPushDownThroughJoin.
// TODO: we will support it in later version.
- /*.peek(expr -> {
- // target is always the expr at the two side of equal of hash conjunctions.
- // TODO: some complex situation cannot be handled now, see testPushDownThroughJoin.
- List<SlotReference> slots = expr.children().stream().filter(SlotReference.class::isInstance)
- .map(SlotReference.class::cast).collect(Collectors.toList());
- if (slots.size() != 2
- || !(ctx.checkExistKey(ctx.getTargetExprIdToFilter(), slots.get(0).getExprId())
- || ctx.checkExistKey(ctx.getTargetExprIdToFilter(), slots.get(1).getExprId()))) {
+ .forEach(expr -> legalTypes.forEach(type -> {
+ Pair<Expression, Expression> normalizedChildren = checkAndMaybeSwapChild(expr, join);
+ // aliasTransMap doesn't contain the key, means that the path from the olap scan to the join
+ // contains join with denied join type. for example: a left join b on a.id = b.id
+ if (normalizedChildren == null
+ || !aliasTransferMap.containsKey((Slot) normalizedChildren.first)) {
return;
}
- int tag = ctx.checkExistKey(ctx.getTargetExprIdToFilter(), slots.get(0).getExprId()) ? 0 : 1;
- // generate runtime filter to associated expr. for example, a = b and a = c, RF b -> a can
- // generate RF b -> c
- List<RuntimeFilter> copiedRuntimeFilter = ctx.getFiltersByTargetExprId(slots.get(tag)
- .getExprId()).stream()
- .map(filter -> new RuntimeFilter(generator.getNextId(), filter.getSrcExpr(),
- slots.get(tag ^ 1), filter.getType(), filter.getExprOrder(), join))
- .collect(Collectors.toList());
- ctx.setTargetExprIdToFilters(slots.get(tag ^ 1).getExprId(),
- copiedRuntimeFilter.toArray(new RuntimeFilter[0]));
- })*/
- .forEach(expr -> legalTypes.stream()
- .map(type -> RuntimeFilter.createRuntimeFilter(generator.getNextId(), expr,
- type, cnt.getAndIncrement(), join))
- .filter(Objects::nonNull)
- .forEach(filter ->
- ctx.setTargetExprIdToFilters(filter.getTargetExpr().getExprId(), filter)));
+ Pair<Slot, Slot> slots = Pair.of(
+ aliasTransferMap.get((Slot) normalizedChildren.first).second.toSlot(),
+ ((Slot) normalizedChildren.second));
+ RuntimeFilter filter = new RuntimeFilter(generator.getNextId(),
+ slots.second, slots.first, type,
+ cnt.getAndIncrement(), join);
+ ctx.setTargetExprIdToFilter(slots.first.getExprId(), filter);
+ ctx.setTargetsOnScanNode(
+ aliasTransferMap.get((Slot) normalizedChildren.first).first,
+ slots.first);
+ }));
}
- join.left().accept(this, context);
- join.right().accept(this, context);
return join;
}
// TODO: support src key is agg slot.
@Override
public PhysicalPlan visitPhysicalProject(PhysicalProject<? extends Plan> project, CascadesContext context) {
- RuntimeFilterContext ctx = context.getRuntimeFilterContext();
+ project.child().accept(this, context);
+ Map<NamedExpression, Pair<RelationId, NamedExpression>> aliasTransferMap
+ = context.getRuntimeFilterContext().getAliasTransferMap();
+ // change key when encounter alias.
project.getProjects().stream().filter(Alias.class::isInstance)
.map(Alias.class::cast)
- .filter(expr -> expr.child() instanceof SlotReference)
- .forEach(expr -> ctx.setKVInNormalMap(ctx.getAliasChildToSelf(), ((SlotReference) expr.child()), expr));
- project.child().accept(this, context);
+ .filter(alias -> alias.child() instanceof NamedExpression
+ && aliasTransferMap.containsKey((NamedExpression) alias.child()))
+ .forEach(alias -> {
+ NamedExpression child = ((NamedExpression) alias.child());
+ aliasTransferMap.put(alias.toSlot(), aliasTransferMap.remove(child));
+ });
return project;
}
@Override
public PhysicalOlapScan visitPhysicalOlapScan(PhysicalOlapScan scan, CascadesContext context) {
+ // add all the slots in map.
RuntimeFilterContext ctx = context.getRuntimeFilterContext();
- scan.getOutput().stream()
- .filter(slot -> ctx.getSlotListOfTheSameSlotAtOlapScanNode(slot).stream()
- .filter(expr -> ctx.checkExistKey(ctx.getTargetExprIdToFilter(), expr.getExprId()))
- .peek(expr -> {
- if (expr.getExprId() == slot.getExprId()) {
- return;
- }
- List<RuntimeFilter> filters = ctx.getFiltersByTargetExprId(expr.getExprId());
- ctx.removeFilters(expr.getExprId());
- filters.forEach(filter -> filter.setTargetSlot(slot));
- ctx.setKVInNormalMap(ctx.getTargetExprIdToFilter(), slot.getExprId(), filters);
- })
- .count() > 0)
- .forEach(slot -> ctx.setTargetsOnScanNode(scan.getId(), slot));
+ scan.getOutput().forEach(slot -> ctx.getAliasTransferMap().put(slot, Pair.of(scan.getId(), slot)));
return scan;
}
+
+ private static Pair<Expression, Expression> checkAndMaybeSwapChild(EqualTo expr,
+ PhysicalHashJoin<? extends Plan, ? extends Plan> join) {
+ if (expr.child(0).equals(expr.child(1))
+ || !expr.children().stream().allMatch(SlotReference.class::isInstance)) {
+ return null;
+ }
+ // current we assume that there are certainly different slot reference in equal to.
+ // they are not from the same relation.
+ List<Expression> children = JoinUtils.swapEqualToForChildrenOrder(expr, join.left().getOutputSet()).children();
+ return Pair.of(children.get(0), children.get(1));
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/RuntimeFilter.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/RuntimeFilter.java
index 8eeb6071d7..326bc6b634 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/RuntimeFilter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/RuntimeFilter.java
@@ -17,12 +17,7 @@
package org.apache.doris.nereids.trees.plans.physical;
-import org.apache.doris.common.Pair;
-import org.apache.doris.nereids.trees.expressions.EqualTo;
-import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
-import org.apache.doris.nereids.trees.expressions.SlotReference;
-import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.planner.RuntimeFilterId;
import org.apache.doris.thrift.TRuntimeFilterType;
@@ -58,37 +53,6 @@ public class RuntimeFilter {
this.builderNode = builderNode;
}
- /**
- * create RF
- */
- public static RuntimeFilter createRuntimeFilter(RuntimeFilterId id, EqualTo conjunction,
- TRuntimeFilterType type, int exprOrder, PhysicalHashJoin node) {
- Pair<Expression, Expression> srcs = checkAndMaybeSwapChild(conjunction, node);
- if (srcs == null) {
- return null;
- }
- return new RuntimeFilter(id, ((SlotReference) srcs.second), ((SlotReference) srcs.first), type, exprOrder,
- node);
- }
-
- private static Pair<Expression, Expression> checkAndMaybeSwapChild(EqualTo expr,
- PhysicalHashJoin join) {
- if (expr.children().stream().anyMatch(Literal.class::isInstance)) {
- return null;
- }
- if (expr.child(0).equals(expr.child(1))) {
- return null;
- }
- if (!expr.children().stream().allMatch(SlotReference.class::isInstance)) {
- return null;
- }
- // current we assume that there are certainly different slot reference in equal to.
- // they are not from the same relation.
- int exchangeTag = join.child(0).getOutput().stream().anyMatch(slot -> slot.getExprId().equals(
- ((SlotReference) expr.child(1)).getExprId())) ? 1 : 0;
- return Pair.of(expr.child(exchangeTag), expr.child(1 ^ exchangeTag));
- }
-
public Slot getSrcExpr() {
return srcSlot;
}
@@ -113,14 +77,6 @@ public class RuntimeFilter {
return builderNode;
}
- public void setTargetSlot(Slot targetSlot) {
- this.targetSlot = targetSlot;
- }
-
- public boolean isUninitialized() {
- return !finalized;
- }
-
public void setFinalized() {
this.finalized = true;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java
index 7b0fe88971..617ccf0f05 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java
@@ -228,9 +228,10 @@ public class RuntimeFilterTest extends SSBTestBase {
private void checkRuntimeFilterExprs(List<RuntimeFilter> filters, List<Pair<String, String>> colNames) {
Assertions.assertEquals(filters.size(), colNames.size());
- for (int i = 0; i < filters.size(); i++) {
- Assertions.assertTrue(filters.get(i).getSrcExpr().toSql().equals(colNames.get(i).first)
- && filters.get(i).getTargetExpr().toSql().equals(colNames.get(i).second));
+ for (RuntimeFilter filter : filters) {
+ Assertions.assertTrue(colNames.contains(Pair.of(
+ filter.getSrcExpr().getName(),
+ filter.getTargetExpr().getName())));
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/39: [feature](Nereids) support query that group by use alias generated in aggregate output (#14030)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5db650f961730e9eb6082dd19cb0905656297961
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Tue Nov 8 11:02:42 2022 +0800
[feature](Nereids) support query that group by use alias generated in aggregate output (#14030)
support query having alias in group by list, such as:
SELECT c1 AS a, SUM(c2) FROM t GROUP BY a;
---
.../nereids/rules/analysis/BindSlotReference.java | 34 ++++++++++++++++------
.../datasets/clickbench/AnalyzeClickBenchTest.java | 24 +++++++--------
2 files changed, 37 insertions(+), 21 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
index 56ef1f887c..07b1763afa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
@@ -27,6 +27,7 @@ import org.apache.doris.nereids.memo.Memo;
import org.apache.doris.nereids.properties.OrderKey;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.UnaryNode;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.Exists;
import org.apache.doris.nereids.trees.expressions.Expression;
@@ -58,6 +59,7 @@ import org.apache.commons.lang.StringUtils;
import java.util.ArrayList;
import java.util.List;
+import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
@@ -67,14 +69,11 @@ import java.util.stream.Stream;
/**
* BindSlotReference.
*/
+@SuppressWarnings("OptionalUsedAsFieldOrParameterType")
public class BindSlotReference implements AnalysisRuleFactory {
private final Optional<Scope> outerScope;
- public BindSlotReference() {
- this(Optional.empty());
- }
-
public BindSlotReference(Optional<Scope> outerScope) {
this.outerScope = Objects.requireNonNull(outerScope, "outerScope cannot be null");
}
@@ -122,10 +121,27 @@ public class BindSlotReference implements AnalysisRuleFactory {
RuleType.BINDING_AGGREGATE_SLOT.build(
logicalAggregate().when(Plan::canBind).thenApply(ctx -> {
LogicalAggregate<GroupPlan> agg = ctx.root;
- List<Expression> groupBy =
- bind(agg.getGroupByExpressions(), agg.children(), agg, ctx.cascadesContext);
List<NamedExpression> output =
bind(agg.getOutputExpressions(), agg.children(), agg, ctx.cascadesContext);
+ final Map<String, Expression> substitutions = output.stream()
+ .filter(ne -> ne instanceof Alias)
+ .map(Alias.class::cast)
+ .collect(Collectors.toMap(Alias::getName, UnaryNode::child));
+ List<Expression> replacedGroupBy = agg.getGroupByExpressions().stream()
+ .map(g -> {
+ if (g instanceof UnboundSlot) {
+ UnboundSlot unboundSlot = (UnboundSlot) g;
+ if (unboundSlot.getNameParts().size() == 1) {
+ String name = unboundSlot.getNameParts().get(0);
+ if (substitutions.containsKey(name)) {
+ return substitutions.get(name);
+ }
+ }
+ }
+ return g;
+ }).collect(Collectors.toList());
+
+ List<Expression> groupBy = bind(replacedGroupBy, agg.children(), agg, ctx.cascadesContext);
return agg.withGroupByAndOutput(groupBy, output);
})
),
@@ -301,14 +317,14 @@ public class BindSlotReference implements AnalysisRuleFactory {
return new BoundStar(getScope().getSlots());
case 1: // select table.*
case 2: // select db.table.*
- return bindQualifiedStar(qualifier, context);
+ return bindQualifiedStar(qualifier);
default:
throw new AnalysisException("Not supported qualifier: "
+ StringUtils.join(qualifier, "."));
}
}
- private BoundStar bindQualifiedStar(List<String> qualifierStar, PlannerContext context) {
+ private BoundStar bindQualifiedStar(List<String> qualifierStar) {
// FIXME: compatible with previous behavior:
// https://github.com/apache/doris/pull/10415/files/3fe9cb0c3f805ab3a9678033b281b16ad93ec60a#r910239452
List<Slot> slots = getScope().getSlots().stream().filter(boundSlot -> {
@@ -544,7 +560,7 @@ public class BindSlotReference implements AnalysisRuleFactory {
public boolean hasGroupBy() {
if (rootIsAgg()) {
- return !((LogicalAggregate) logicalPlan).getGroupByExpressions().isEmpty();
+ return !((LogicalAggregate<? extends Plan>) logicalPlan).getGroupByExpressions().isEmpty();
}
return false;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
index 0493582d74..65a18b8591 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
@@ -110,10 +110,10 @@ public class AnalyzeClickBenchTest extends ClickBenchTestBase {
checkAnalyze(ClickBenchUtils.Q17);
}
- // @Test
- // public void q18() {
- // checkAnalyze(ClickBenchUtils.Q18);
- // }
+ @Test
+ public void q18() {
+ checkAnalyze(ClickBenchUtils.Q18);
+ }
@Test
public void q19() {
@@ -160,10 +160,10 @@ public class AnalyzeClickBenchTest extends ClickBenchTestBase {
checkAnalyze(ClickBenchUtils.Q27);
}
- // @Test
- // public void q28() {
- // checkAnalyze(ClickBenchUtils.Q28);
- // }
+ @Test
+ public void q28() {
+ checkAnalyze(ClickBenchUtils.Q28);
+ }
@Test
public void q29() {
@@ -215,10 +215,10 @@ public class AnalyzeClickBenchTest extends ClickBenchTestBase {
checkAnalyze(ClickBenchUtils.Q38);
}
- // @Test
- // public void q39() {
- // checkAnalyze(ClickBenchUtils.Q39);
- // }
+ @Test
+ public void q39() {
+ checkAnalyze(ClickBenchUtils.Q39);
+ }
@Test
public void q40() {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 27/39: [bug](ColumnDecimal)call set_decimalv2_type when cloning ColumnDecimal (#14061)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b5db5c1f0a7bebc06745949afe394eaf6a17c34c
Author: Kang <kx...@gmail.com>
AuthorDate: Wed Nov 9 11:23:43 2022 +0800
[bug](ColumnDecimal)call set_decimalv2_type when cloning ColumnDecimal (#14061)
* call set_decimalv2_type when cloning ColumnDecimal
* clang format
---
be/src/vec/columns/column_decimal.cpp | 3 ++
be/test/CMakeLists.txt | 1 +
be/test/vec/columns/column_decimal_test.cpp | 53 +++++++++++++++++++++++++++++
3 files changed, 57 insertions(+)
diff --git a/be/src/vec/columns/column_decimal.cpp b/be/src/vec/columns/column_decimal.cpp
index 078494608c..8194b1ff7e 100644
--- a/be/src/vec/columns/column_decimal.cpp
+++ b/be/src/vec/columns/column_decimal.cpp
@@ -214,6 +214,9 @@ ColumnPtr ColumnDecimal<T>::permute(const IColumn::Permutation& perm, size_t lim
template <typename T>
MutableColumnPtr ColumnDecimal<T>::clone_resized(size_t size) const {
auto res = this->create(0, scale);
+ if (this->is_decimalv2_type()) {
+ res->set_decimalv2_type();
+ }
if (size > 0) {
auto& new_col = assert_cast<Self&>(*res);
diff --git a/be/test/CMakeLists.txt b/be/test/CMakeLists.txt
index de353b0c6d..85fbb1851d 100644
--- a/be/test/CMakeLists.txt
+++ b/be/test/CMakeLists.txt
@@ -331,6 +331,7 @@ set(VEC_TEST_FILES
vec/aggregate_functions/vec_window_funnel_test.cpp
vec/aggregate_functions/vec_retention_test.cpp
vec/aggregate_functions/agg_min_max_by_test.cpp
+ vec/columns/column_decimal_test.cpp
vec/core/block_test.cpp
vec/core/column_array_test.cpp
vec/core/column_complex_test.cpp
diff --git a/be/test/vec/columns/column_decimal_test.cpp b/be/test/vec/columns/column_decimal_test.cpp
new file mode 100644
index 0000000000..c9f79ee8a0
--- /dev/null
+++ b/be/test/vec/columns/column_decimal_test.cpp
@@ -0,0 +1,53 @@
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/columns/column_decimal.h"
+
+#include <gtest/gtest.h>
+
+#include <memory>
+#include <string>
+
+#include "vec/columns/columns_number.h"
+
+namespace doris::vectorized {
+
+TEST(ColumnDecimalTest, CloneDecimalv2TypeTest) {
+ auto column1 = ColumnDecimal128::create(0, 1);
+ EXPECT_EQ(false, column1->is_decimalv2_type());
+ EXPECT_EQ(false, column1->clone_resized(0)->is_decimalv2_type());
+ column1->set_decimalv2_type();
+ EXPECT_EQ(true, column1->is_decimalv2_type());
+ EXPECT_EQ(true, column1->clone_resized(0)->is_decimalv2_type());
+
+ auto column2 = ColumnDecimal64::create(0, 1);
+ EXPECT_EQ(false, column2->is_decimalv2_type());
+ EXPECT_EQ(false, column2->clone_resized(0)->is_decimalv2_type());
+ column2->set_decimalv2_type();
+ EXPECT_EQ(true, column2->is_decimalv2_type());
+ EXPECT_EQ(true, column2->clone_resized(0)->is_decimalv2_type());
+
+ auto column3 = ColumnDecimal32::create(0, 1);
+ EXPECT_EQ(false, column3->is_decimalv2_type());
+ EXPECT_EQ(false, column3->clone_resized(0)->is_decimalv2_type());
+ column3->set_decimalv2_type();
+ EXPECT_EQ(true, column3->is_decimalv2_type());
+ EXPECT_EQ(true, column3->clone_resized(0)->is_decimalv2_type());
+}
+
+} // namespace doris::vectorized
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 12/39: [chore](bin) do not set heap limit for tcmalloc until doris does not allocates large unused memory (#13761)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2367e098d241f675bbc8faf3da1ab2a6cc53c1b9
Author: Yongqiang YANG <98...@users.noreply.github.com>
AuthorDate: Tue Nov 8 19:26:30 2022 +0800
[chore](bin) do not set heap limit for tcmalloc until doris does not allocates large unused memory (#13761)
We set heap limit for tcmalloc to avoid oom introduced by tcmalloc which allocates memory for cache even free memory of a machine is little. However, doris allocates large memory unused in some cases, so tcmalloc would throw an oom exception even ther are a lot free memory in a machine.
We can set the limit after we fix the problem again.
---
bin/start_be.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index cdd051bf7d..81ec80da87 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -220,7 +220,7 @@ set_tcmalloc_heap_limit() {
export TCMALLOC_HEAP_LIMIT_MB=${mem_limit_mb}
}
-set_tcmalloc_heap_limit || exit 1
+# set_tcmalloc_heap_limit || exit 1
## set hdfs conf
export LIBHDFS3_CONF="${DORIS_HOME}/conf/hdfs-site.xml"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 36/39: [feature](table-valued-function) Support `desc from s3()` and modify the syntax of tvf (#14047)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 09650f22e517a191f84e3d2e4a08826364c4e8ce
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Wed Nov 9 14:12:43 2022 +0800
[feature](table-valued-function) Support `desc from s3()` and modify the syntax of tvf (#14047)
This pr does two things:
Support desc function s3()
modify the syntax of tvf
---
.../sql-functions/table-functions/numbers.md | 11 +-
.../sql-functions/table-functions/numbers.md | 11 +-
fe/fe-core/src/main/cup/sql_parser.cup | 8 +-
.../org/apache/doris/analysis/DescribeStmt.java | 46 +++++--
.../doris/analysis/TableValuedFunctionRef.java | 4 +-
.../tablefunction/NumbersTableValuedFunction.java | 45 +++++--
.../doris/tablefunction/S3TableValuedFunction.java | 39 ++++--
.../doris/tablefunction/TableValuedFunctionIf.java | 4 +-
.../table_valued_function/test_numbers.groovy | 134 ++++++++++-----------
.../http_rest_api/post/test_query_stmt.groovy | 2 +-
.../datetime_functions/test_date_function.groovy | 2 +-
11 files changed, 188 insertions(+), 118 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-functions/table-functions/numbers.md b/docs/en/docs/sql-manual/sql-functions/table-functions/numbers.md
index ca0f5de915..2d7d034cf3 100644
--- a/docs/en/docs/sql-manual/sql-functions/table-functions/numbers.md
+++ b/docs/en/docs/sql-manual/sql-functions/table-functions/numbers.md
@@ -34,16 +34,19 @@ This function is used in FROM clauses.
grammar:
```
-FROM numbers(n[,m]);
+numbers(
+ "number" = "n",
+ "backend_num" = "m"
+ );
```
parameter:
-- `n`: It means to generate rows [0, n).
-- `m`: Optional parameters. It means this function is executed simultaneously on `m` be nodes (multiple BEs need to be deployed).
+- `number`: It means to generate rows [0, n).
+- `backend_num`: Optional parameters. It means this function is executed simultaneously on `m` be nodes (multiple BEs need to be deployed).
### example
```
-mysql> select * from numbers("5");
+mysql> select * from numbers("number" = "10");
+--------+
| number |
+--------+
diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/numbers.md b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/numbers.md
index 60605f7282..a5a0162ed4 100644
--- a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/numbers.md
+++ b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/numbers.md
@@ -35,16 +35,19 @@ under the License.
语法:
```
-FROM numbers(n[,m]);
+numbers(
+ "number" = "n",
+ "backend_num" = "m"
+ );
```
参数:
-- `n`: 代表生成[0,n)的行。
-- `m`: 可选参数,代表`m`个be节点同时执行该函数(需要部署多个be)。
+- `number`: 代表生成[0,n)的行。
+- `backend_num`: 可选参数,代表`m`个be节点同时执行该函数(需要部署多个be)。
### example
```
-mysql> select * from numbers("10");
+mysql> select * from numbers("number" = "10");
+--------+
| number |
+--------+
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup b/fe/fe-core/src/main/cup/sql_parser.cup
index adec23db2c..587b7f865d 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -3818,6 +3818,10 @@ describe_stmt ::=
{:
RESULT = new DescribeStmt(table, false);
:}
+ | describe_command KW_FUNCTION table_valued_function_ref:tvf
+ {:
+ RESULT = new DescribeStmt(tvf);
+ :}
| describe_command table_name:table KW_ALL
{:
RESULT = new DescribeStmt(table, true);
@@ -4962,9 +4966,9 @@ table_ref ::=
;
table_valued_function_ref ::=
- ident:func_name LPAREN string_list:param_list RPAREN opt_table_alias:alias
+ ident:func_name LPAREN key_value_map:properties RPAREN opt_table_alias:alias
{:
- RESULT = new TableValuedFunctionRef(func_name, alias, param_list);
+ RESULT = new TableValuedFunctionRef(func_name, alias, properties);
:}
;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DescribeStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DescribeStmt.java
index 8e481da21c..c76db30c8a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DescribeStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DescribeStmt.java
@@ -33,8 +33,8 @@ import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.proc.IndexSchemaProcNode;
import org.apache.doris.common.proc.ProcNodeInterface;
-import org.apache.doris.common.proc.ProcResult;
import org.apache.doris.common.proc.ProcService;
import org.apache.doris.common.proc.TableProcDir;
import org.apache.doris.common.util.Util;
@@ -84,23 +84,48 @@ public class DescribeStmt extends ShowStmt {
private TableName dbTableName;
private ProcNodeInterface node;
- List<List<String>> totalRows;
+ List<List<String>> totalRows = new LinkedList<List<String>>();
private boolean isAllTables;
- private boolean isOlapTable;
+ private boolean isOlapTable = false;
+
+ TableValuedFunctionRef tableValuedFunctionRef;
+ boolean isTableValuedFunction;
public DescribeStmt(TableName dbTableName, boolean isAllTables) {
this.dbTableName = dbTableName;
- this.totalRows = new LinkedList<List<String>>();
this.isAllTables = isAllTables;
}
+ public DescribeStmt(TableValuedFunctionRef tableValuedFunctionRef) {
+ this.tableValuedFunctionRef = tableValuedFunctionRef;
+ this.isTableValuedFunction = true;
+ this.isAllTables = false;
+ }
+
public boolean isAllTables() {
return isAllTables;
}
@Override
public void analyze(Analyzer analyzer) throws UserException {
+ if (!isAllTables && isTableValuedFunction) {
+ List<Column> columns = tableValuedFunctionRef.getTableFunction().getTable().getBaseSchema();
+ for (Column column : columns) {
+ List<String> row = Arrays.asList(
+ column.getDisplayName(),
+ column.getOriginType().toString(),
+ column.isAllowNull() ? "Yes" : "No",
+ ((Boolean) column.isKey()).toString(),
+ column.getDefaultValue() == null
+ ? FeConstants.null_string : column.getDefaultValue(),
+ "NONE"
+ );
+ totalRows.add(row);
+ }
+ return;
+ }
+
dbTableName.analyze(analyzer);
if (!Env.getCurrentEnv().getAuth().checkTblPriv(ConnectContext.get(), dbTableName, PrivPredicate.SHOW)) {
@@ -241,6 +266,9 @@ public class DescribeStmt extends ShowStmt {
if (isAllTables) {
return totalRows;
} else {
+ if (isTableValuedFunction) {
+ return totalRows;
+ }
Preconditions.checkNotNull(node);
return node.fetchResult().getRows();
}
@@ -250,15 +278,7 @@ public class DescribeStmt extends ShowStmt {
public ShowResultSetMetaData getMetaData() {
if (!isAllTables) {
ShowResultSetMetaData.Builder builder = ShowResultSetMetaData.builder();
-
- ProcResult result = null;
- try {
- result = node.fetchResult();
- } catch (AnalysisException e) {
- return builder.build();
- }
-
- for (String col : result.getColumnNames()) {
+ for (String col : IndexSchemaProcNode.TITLE_NAMES) {
builder.addColumn(new Column(col, ScalarType.createVarchar(30)));
}
return builder.build();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TableValuedFunctionRef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/TableValuedFunctionRef.java
index a697e68fef..328b1819fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TableValuedFunctionRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TableValuedFunctionRef.java
@@ -23,14 +23,14 @@ import org.apache.doris.planner.PlanNodeId;
import org.apache.doris.planner.ScanNode;
import org.apache.doris.tablefunction.TableValuedFunctionIf;
-import java.util.List;
+import java.util.Map;
public class TableValuedFunctionRef extends TableRef {
private Table table;
private TableValuedFunctionIf tableFunction;
- public TableValuedFunctionRef(String funcName, String alias, List<String> params) throws UserException {
+ public TableValuedFunctionRef(String funcName, String alias, Map<String, String> params) throws UserException {
super(new TableName(null, null, "_table_valued_function_" + funcName), alias);
this.tableFunction = TableValuedFunctionIf.getTableFunction(funcName, params);
if (hasExplicitAlias()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/NumbersTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/NumbersTableValuedFunction.java
index 3be38e7a0f..5b334dbda9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/NumbersTableValuedFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/NumbersTableValuedFunction.java
@@ -32,20 +32,30 @@ import org.apache.doris.thrift.TDataGenScanRange;
import org.apache.doris.thrift.TScanRange;
import org.apache.doris.thrift.TTVFNumbersScanRange;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
+import java.util.Map;
// Table function that generate int64 numbers
// have a single column number
/**
- * The Implement of table valued function——numbers(N,M).
+ * The Implement of table valued function——numbers("number" = "N", "backend_num" = "M").
*/
public class NumbersTableValuedFunction extends DataGenTableValuedFunction {
public static final String NAME = "numbers";
+ public static final String NUMBER = "number";
+ public static final String BACKEND_NUM = "backend_num";
+ private static final ImmutableSet<String> PROPERTIES_SET = new ImmutableSet.Builder<String>()
+ .add(NUMBER)
+ .add(BACKEND_NUM)
+ .build();
// The total numbers will be generated.
private long totalNumbers;
// The total backends will server it.
@@ -56,17 +66,30 @@ public class NumbersTableValuedFunction extends DataGenTableValuedFunction {
* @param params params from user
* @throws UserException exception
*/
- public NumbersTableValuedFunction(List<String> params) throws UserException {
- if (params.size() < 1 || params.size() > 2) {
- throw new UserException(
- "numbers table function only support numbers(10000 /*total numbers*/)"
- + "or numbers(10000, 2 /*number of tablets to run*/)");
+ public NumbersTableValuedFunction(Map<String, String> params) throws UserException {
+ Map<String, String> validParams = Maps.newHashMap();
+ for (String key : params.keySet()) {
+ if (!PROPERTIES_SET.contains(key.toLowerCase())) {
+ throw new AnalysisException(key + " is invalid property");
+ }
+ validParams.put(key.toLowerCase(), params.get(key));
+ }
+
+ try {
+ tabletsNum = Integer.parseInt(validParams.getOrDefault(BACKEND_NUM, "1"));
+ } catch (NumberFormatException e) {
+ throw new UserException("can not parse `backend_num` param to natural number");
}
- totalNumbers = Long.parseLong(params.get(0));
- // default tabletsNum is 1.
- tabletsNum = 1;
- if (params.size() == 2) {
- tabletsNum = Integer.parseInt(params.get(1));
+ String numberStr = validParams.get(NUMBER);
+ if (!Strings.isNullOrEmpty(numberStr)) {
+ try {
+ totalNumbers = Long.parseLong(numberStr);
+ } catch (NumberFormatException e) {
+ throw new UserException("can not parse `number` param to natural number");
+ }
+ } else {
+ throw new UserException(
+ "can not find `number` param, please specify `number`, like: numbers(\"number\" = \"10\")");
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/S3TableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/S3TableValuedFunction.java
index 19655fbedd..1a615e787c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/S3TableValuedFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/S3TableValuedFunction.java
@@ -26,40 +26,55 @@ import org.apache.doris.common.util.S3URI;
import org.apache.doris.thrift.TFileFormatType;
import org.apache.doris.thrift.TFileType;
+import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
-import java.util.List;
+import java.util.Map;
/**
- * The Implement of table valued function——S3(path, AK, SK, format).
+ * The Implement of table valued function
+ * S3("uri" = "xxx", "access_key" = "xx", "SECRET_KEY" = "qqq", "FORMAT" = "csv").
*/
public class S3TableValuedFunction extends ExternalFileTableValuedFunction {
public static final Logger LOG = LogManager.getLogger(S3TableValuedFunction.class);
public static final String NAME = "s3";
+ public static final String S3_URI = "URI";
public static final String S3_AK = "AWS_ACCESS_KEY";
public static final String S3_SK = "AWS_SECRET_KEY";
public static final String S3_ENDPOINT = "AWS_ENDPOINT";
public static final String S3_REGION = "AWS_REGION";
+ public static final String FORMAT = "FORMAT";
+ private static final String AK = "ACCESS_KEY";
+ private static final String SK = "SECRET_KEY";
+
public static final String USE_PATH_STYLE = "use_path_style";
+ private static final ImmutableSet<String> PROPERTIES_SET = new ImmutableSet.Builder<String>()
+ .add(S3_URI)
+ .add(AK)
+ .add(SK)
+ .add(FORMAT)
+ .build();
private S3URI s3uri;
private String s3AK;
private String s3SK;
- public S3TableValuedFunction(List<String> params) throws UserException {
- if (params.size() != 4) {
- throw new UserException(
- "s3 table function only support 4 params now: S3(path, AK, SK, format)");
+ public S3TableValuedFunction(Map<String, String> params) throws UserException {
+ Map<String, String> validParams = Maps.newHashMap();
+ for (String key : params.keySet()) {
+ if (!PROPERTIES_SET.contains(key.toUpperCase())) {
+ throw new AnalysisException(key + " is invalid property");
+ }
+ validParams.put(key.toUpperCase(), params.get(key));
}
- s3uri = S3URI.create(params.get(0));
- s3AK = params.get(1);
- s3SK = params.get(2);
-
- String formatString = params.get(3).toLowerCase();
- switch (formatString) {
+ s3uri = S3URI.create(validParams.get(S3_URI));
+ s3AK = validParams.getOrDefault(AK, "");
+ s3SK = validParams.getOrDefault(SK, "");
+ String formatString = validParams.getOrDefault(FORMAT, "");
+ switch (formatString.toLowerCase()) {
case "csv":
this.fileFormatType = TFileFormatType.FORMAT_CSV_PLAIN;
break;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
index 09ef70ccb5..3063880c92 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
@@ -27,6 +27,7 @@ import org.apache.doris.planner.PlanNodeId;
import org.apache.doris.planner.ScanNode;
import java.util.List;
+import java.util.Map;
public abstract class TableValuedFunctionIf {
private FunctionGenTable table = null;
@@ -40,7 +41,8 @@ public abstract class TableValuedFunctionIf {
}
// All table functions should be registered here
- public static TableValuedFunctionIf getTableFunction(String funcName, List<String> params) throws UserException {
+ public static TableValuedFunctionIf getTableFunction(String funcName, Map<String, String> params)
+ throws UserException {
switch (funcName.toLowerCase()) {
case NumbersTableValuedFunction.NAME:
return new NumbersTableValuedFunction(params);
diff --git a/regression-test/suites/correctness_p0/table_valued_function/test_numbers.groovy b/regression-test/suites/correctness_p0/table_valued_function/test_numbers.groovy
index a70e25e6c3..fd2bc9b444 100644
--- a/regression-test/suites/correctness_p0/table_valued_function/test_numbers.groovy
+++ b/regression-test/suites/correctness_p0/table_valued_function/test_numbers.groovy
@@ -18,105 +18,105 @@
suite("test_numbers") {
// Test basic features
- qt_basic1 """ select * from numbers("1"); """
- qt_basic2 """ select * from numbers("10"); """
- qt_basic3 """ select * from numbers("100"); """
- qt_basic4_limit """ select * from numbers("10") limit 5; """
+ qt_basic1 """ select * from numbers("number" = "1"); """
+ qt_basic2 """ select * from numbers("number" = "10"); """
+ qt_basic3 """ select * from numbers("number" = "100"); """
+ qt_basic4_limit """ select * from numbers("number" = "10") limit 5; """
- // Test aggregate function withh numbers(N)
- qt_agg_sum """ select sum(number) from numbers("100"); """
- qt_agg_avg """ select avg(number) from numbers("100"); """
- qt_agg_count """ select count(*) from numbers("100"); """
- qt_agg_min """ select min(number) from numbers("100"); """
- qt_agg_max """ select max(number) from numbers("100"); """
+ // Test aggregate function withh numbers("number" = N)
+ qt_agg_sum """ select sum(number) from numbers("number" = "100"); """
+ qt_agg_avg """ select avg(number) from numbers("number" = "100"); """
+ qt_agg_count """ select count(*) from numbers("number" = "100"); """
+ qt_agg_min """ select min(number) from numbers("number" = "100"); """
+ qt_agg_max """ select max(number) from numbers("number" = "100"); """
- // Test join with numbers(N)
+ // Test join with numbers("number" = N)
qt_inner_join1 """
select a.number as num1, b.number as num2
- from numbers("10") a inner join numbers("10") b
+ from numbers("number" = "10") a inner join numbers("number" = "10") b
on a.number=b.number;
"""
qt_inner_join2 """
select a.number as num1, b.number as num2
- from numbers("6") a inner join numbers("6") b
+ from numbers("number" = "6") a inner join numbers("number" = "6") b
on a.number>b.number;
"""
qt_inner_join3 """
select a.number as num1, b.number as num2
- from numbers("10") a inner join numbers("10") b
+ from numbers("number" = "10") a inner join numbers("number" = "10") b
on a.number=b.number and b.number%2 = 0;
"""
qt_left_join """
select a.number as num1, b.number as num2
- from numbers("10") a left join numbers("5") b
+ from numbers("number" = "10") a left join numbers("number" = "5") b
on a.number=b.number order by num1;
"""
qt_right_join """
select a.number as num1, b.number as num2
- from numbers("5") a right join numbers("10") b
+ from numbers("number" = "5") a right join numbers("number" = "10") b
on a.number=b.number order by num2;
"""
// Test where and GroupBy
- qt_where_equal """ select * from numbers("10") where number%2 = 1; """
- qt_where_gt """ select * from numbers("10") where number-1 > 1; """
- qt_where_lt """ select * from numbers("10") where number+1 < 9; """
- qt_groupby """ select number from numbers("10") where number>=4 group by number order by number; """
+ qt_where_equal """ select * from numbers("number" = "10") where number%2 = 1; """
+ qt_where_gt """ select * from numbers("number" = "10") where number-1 > 1; """
+ qt_where_lt """ select * from numbers("number" = "10") where number+1 < 9; """
+ qt_groupby """ select number from numbers("number" = "10") where number>=4 group by number order by number; """
qt_join_where """
select a.number as num1, b.number as num2
- from numbers("10") a inner join numbers("10") b
+ from numbers("number" = "10") a inner join numbers("number" = "10") b
on a.number=b.number where a.number>4;
"""
// Test Sub Query
- qt_subquery1 """ select * from numbers("10") where number = (select number from numbers("10") where number=1); """
- qt_subquery2 """ select * from numbers("10") where number in (select number from numbers("10") where number>5); """
- qt_subquery3 """ select a.number from numbers("10") a where number in (select number from numbers("10") b where a.number=b.number); """
+ qt_subquery1 """ select * from numbers("number" = "10") where number = (select number from numbers("number" = "10") where number=1); """
+ qt_subquery2 """ select * from numbers("number" = "10") where number in (select number from numbers("number" = "10") where number>5); """
+ qt_subquery3 """ select a.number from numbers("number" = "10") a where number in (select number from numbers("number" = "10") b where a.number=b.number); """
// Test window function
- qt_window_1 """ SELECT row_number() OVER (ORDER BY number) AS id,number from numbers("10"); """
- qt_window_2 """ SELECT number, rank() OVER (order by number) AS sum_three from numbers("10"); """
- qt_window_3 """ SELECT number, dense_rank() OVER (order by number) AS sum_three from numbers("10"); """
- qt_window_4 """ SELECT number, sum(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("10"); """
- qt_window_5 """ SELECT number, min(number) OVER (ORDER BY number rows between 1 PRECEDING and 1 following) AS result from numbers("10"); """
- qt_window_6 """ SELECT number, min(number) OVER (ORDER BY number rows between UNBOUNDED PRECEDING and 1 following) AS result from numbers("10"); """
- qt_window_7 """ SELECT number, max(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("10"); """
- qt_window_8 """ SELECT number, max(number) OVER (ORDER BY number rows between UNBOUNDED PRECEDING and 1 following) AS result from numbers("10"); """
- qt_window_9 """ SELECT number, avg(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("10"); """
- qt_window_10 """ SELECT number, count(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("10"); """
- qt_window_11 """ SELECT number, first_value(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("10"); """
- qt_window_12 """ SELECT number, last_value(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("10"); """
- qt_window_13 """ SELECT number, LAG(number,2,-1) OVER (ORDER BY number) AS result from numbers("10"); """
+ qt_window_1 """ SELECT row_number() OVER (ORDER BY number) AS id,number from numbers("number" = "10"); """
+ qt_window_2 """ SELECT number, rank() OVER (order by number) AS sum_three from numbers("number" = "10"); """
+ qt_window_3 """ SELECT number, dense_rank() OVER (order by number) AS sum_three from numbers("number" = "10"); """
+ qt_window_4 """ SELECT number, sum(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_5 """ SELECT number, min(number) OVER (ORDER BY number rows between 1 PRECEDING and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_6 """ SELECT number, min(number) OVER (ORDER BY number rows between UNBOUNDED PRECEDING and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_7 """ SELECT number, max(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_8 """ SELECT number, max(number) OVER (ORDER BY number rows between UNBOUNDED PRECEDING and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_9 """ SELECT number, avg(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_10 """ SELECT number, count(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_11 """ SELECT number, first_value(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_12 """ SELECT number, last_value(number) OVER (ORDER BY number rows between 1 preceding and 1 following) AS result from numbers("number" = "10"); """
+ qt_window_13 """ SELECT number, LAG(number,2,-1) OVER (ORDER BY number) AS result from numbers("number" = "10"); """
// Cast BITINT to STRING and test string function.
- qt_stringfunction_1 """ select cast (number as string) as string_num from numbers("10"); """
- qt_stringfunction_2 """ select append_trailing_char_if_absent(cast (number as string),'a') as string_fucntion_res from numbers("10"); """
- qt_stringfunction_3 """ select concat(cast (number as string),'abc','d') as string_fucntion_res from numbers("10"); """
- qt_stringfunction_4 """ select concat(cast (number as string), cast (number as string)) as string_fucntion_res from numbers("10"); """
- qt_stringfunction_5 """ select ascii(cast (number as string)) as string_fucntion_res from numbers("12"); """
- qt_stringfunction_6 """ select bit_length(cast (number as string)) as string_fucntion_res from numbers("14") where number>5; """
- qt_stringfunction_7 """ select char_length(cast (number as string)) as string_fucntion_res from numbers("14") where number>5; """
- qt_stringfunction_8 """ select concat_ws('-',cast (number as string),'a') as string_fucntion_res from numbers("14") where number>5; """
- qt_stringfunction_9 """ select number, ends_with(cast (number as string),'1') as string_fucntion_res from numbers("12"); """
- qt_stringfunction_10 """ select number,find_in_set(cast (number as string),'0,1,2,3,4,5,6,7') as string_fucntion_res from numbers("10"); """
- qt_stringfunction_11 """ select number,hex(number) as string_fucntion_res from numbers("13") where number>5; """
- qt_stringfunction_12 """ select number,hex(cast (number as string)) as string_fucntion_res from numbers("13") where number>5; """
- qt_stringfunction_13 """ select number,instr(cast (number as string),'1') as string_fucntion_res from numbers("13") where number>5; """
- qt_stringfunction_14 """ select number,left(cast (number as string),'2') as string_fucntion_res from numbers("1000") where number>120 limit 10; """
- qt_stringfunction_15 """ select number,length(cast (number as string)) as string_fucntion_res from numbers("1000") where number>120 limit 10; """
- qt_stringfunction_16 """ select number,locate('2',cast (number as string)) as string_fucntion_res from numbers("1000") where number>120 limit 10; """
- qt_stringfunction_17 """ select number,locate('2',cast (number as string),3) as string_fucntion_res from numbers("1000") where number>120 limit 10; """
- qt_stringfunction_18 """ select number,lpad(cast (number as string),3,'0') as string_fucntion_res from numbers("1000") where number>95 limit 15; """
- qt_stringfunction_19 """ select ltrim( concat(' a',cast (number as string))) as string_fucntion_res from numbers("10"); """
- qt_stringfunction_20 """ select repeat(cast (number as string),2) as string_fucntion_res from numbers("13"); """
- qt_stringfunction_21 """ select replace(cast (number as string),'1','a') as string_fucntion_res from numbers("13"); """
- qt_stringfunction_22 """ select reverse(cast (number as string)) as string_fucntion_res from numbers("20") where number>9; """
- qt_stringfunction_23 """ select right(cast (number as string),1) as string_fucntion_res from numbers("20") where number>9; """
- qt_stringfunction_24 """ select number,rpad(cast (number as string),3,'0') as string_fucntion_res from numbers("1000") where number>95 limit 15; """
- qt_stringfunction_25 """ select STARTS_WITH(cast (number as string),'1') as string_fucntion_res from numbers("15"); """
- qt_stringfunction_26 """ select strleft(cast (number as string),'2') as string_fucntion_res from numbers("200") where number>105 limit 10; """
- qt_stringfunction_27 """ select strright(cast (number as string),'2') as string_fucntion_res from numbers("1000") where number>105 limit 10; """
- qt_stringfunction_28 """ select substring(cast (number as string),2) as string_fucntion_res from numbers("1000") where number>105 limit 10; """
- qt_stringfunction_29 """ select substring(cast (number as string),-1) as string_fucntion_res from numbers("1000") where number>105 limit 10; """
- qt_stringfunction_30 """ select number,unhex(cast (number as string)) as string_fucntion_res from numbers("100") limit 30; """
+ qt_stringfunction_1 """ select cast (number as string) as string_num from numbers("number" = "10"); """
+ qt_stringfunction_2 """ select append_trailing_char_if_absent(cast (number as string),'a') as string_fucntion_res from numbers("number" = "10"); """
+ qt_stringfunction_3 """ select concat(cast (number as string),'abc','d') as string_fucntion_res from numbers("number" = "10"); """
+ qt_stringfunction_4 """ select concat(cast (number as string), cast (number as string)) as string_fucntion_res from numbers("number" = "10"); """
+ qt_stringfunction_5 """ select ascii(cast (number as string)) as string_fucntion_res from numbers("number" = "12"); """
+ qt_stringfunction_6 """ select bit_length(cast (number as string)) as string_fucntion_res from numbers("number" = "14") where number>5; """
+ qt_stringfunction_7 """ select char_length(cast (number as string)) as string_fucntion_res from numbers("number" = "14") where number>5; """
+ qt_stringfunction_8 """ select concat_ws('-',cast (number as string),'a') as string_fucntion_res from numbers("number" = "14") where number>5; """
+ qt_stringfunction_9 """ select number, ends_with(cast (number as string),'1') as string_fucntion_res from numbers("number" = "12"); """
+ qt_stringfunction_10 """ select number,find_in_set(cast (number as string),'0,1,2,3,4,5,6,7') as string_fucntion_res from numbers("number" = "10"); """
+ qt_stringfunction_11 """ select number,hex(number) as string_fucntion_res from numbers("number" = "13") where number>5; """
+ qt_stringfunction_12 """ select number,hex(cast (number as string)) as string_fucntion_res from numbers("number" = "13") where number>5; """
+ qt_stringfunction_13 """ select number,instr(cast (number as string),'1') as string_fucntion_res from numbers("number" = "13") where number>5; """
+ qt_stringfunction_14 """ select number,left(cast (number as string),'2') as string_fucntion_res from numbers("number" = "1000") where number>120 limit 10; """
+ qt_stringfunction_15 """ select number,length(cast (number as string)) as string_fucntion_res from numbers("number" = "1000") where number>120 limit 10; """
+ qt_stringfunction_16 """ select number,locate('2',cast (number as string)) as string_fucntion_res from numbers("number" = "1000") where number>120 limit 10; """
+ qt_stringfunction_17 """ select number,locate('2',cast (number as string),3) as string_fucntion_res from numbers("number" = "1000") where number>120 limit 10; """
+ qt_stringfunction_18 """ select number,lpad(cast (number as string),3,'0') as string_fucntion_res from numbers("number" = "1000") where number>95 limit 15; """
+ qt_stringfunction_19 """ select ltrim( concat(' a',cast (number as string))) as string_fucntion_res from numbers("number" = "10"); """
+ qt_stringfunction_20 """ select repeat(cast (number as string),2) as string_fucntion_res from numbers("number" = "13"); """
+ qt_stringfunction_21 """ select replace(cast (number as string),'1','a') as string_fucntion_res from numbers("number" = "13"); """
+ qt_stringfunction_22 """ select reverse(cast (number as string)) as string_fucntion_res from numbers("number" = "20") where number>9; """
+ qt_stringfunction_23 """ select right(cast (number as string),1) as string_fucntion_res from numbers("number" = "20") where number>9; """
+ qt_stringfunction_24 """ select number,rpad(cast (number as string),3,'0') as string_fucntion_res from numbers("number" = "1000") where number>95 limit 15; """
+ qt_stringfunction_25 """ select STARTS_WITH(cast (number as string),'1') as string_fucntion_res from numbers("number" = "15"); """
+ qt_stringfunction_26 """ select strleft(cast (number as string),'2') as string_fucntion_res from numbers("number" = "200") where number>105 limit 10; """
+ qt_stringfunction_27 """ select strright(cast (number as string),'2') as string_fucntion_res from numbers("number" = "1000") where number>105 limit 10; """
+ qt_stringfunction_28 """ select substring(cast (number as string),2) as string_fucntion_res from numbers("number" = "1000") where number>105 limit 10; """
+ qt_stringfunction_29 """ select substring(cast (number as string),-1) as string_fucntion_res from numbers("number" = "1000") where number>105 limit 10; """
+ qt_stringfunction_30 """ select number,unhex(cast (number as string)) as string_fucntion_res from numbers("number" = "100") limit 30; """
}
diff --git a/regression-test/suites/http_rest_api/post/test_query_stmt.groovy b/regression-test/suites/http_rest_api/post/test_query_stmt.groovy
index 00bc133b6b..7a03cdc48b 100644
--- a/regression-test/suites/http_rest_api/post/test_query_stmt.groovy
+++ b/regression-test/suites/http_rest_api/post/test_query_stmt.groovy
@@ -49,7 +49,7 @@ suite("test_query_stmt") {
def url= "/api/query/default_cluster/" + context.config.defaultDb
// test select
- def stmt1 = """ select * from numbers('10', '1') """
+ def stmt1 = """ select * from numbers('number' = '10', 'backend_num' = '1') """
def stmt1_json = JsonOutput.toJson(new Stmt(stmt: stmt1));
def resJson = http_post(url, stmt1_json)
diff --git a/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy b/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
index b63de704e1..e9544a3983 100644
--- a/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
+++ b/regression-test/suites/query_p0/sql_functions/datetime_functions/test_date_function.groovy
@@ -263,7 +263,7 @@ suite("test_date_function") {
qt_sql """ select yearweek('1989-03-21', 6) """
qt_sql """ select yearweek('1989-03-21', 7) """
- qt_sql """ select count(*) from (select * from numbers("200")) tmp1 WHERE 0 <= UNIX_TIMESTAMP(); """
+ qt_sql """ select count(*) from (select * from numbers("number" = "200")) tmp1 WHERE 0 <= UNIX_TIMESTAMP(); """
sql """ drop table ${tableName} """
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 38/39: [refractor](array) refractor DataTypeArray from_string (#13905)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6f022b40a2e33b23fec48d36abc66621daa1e2f1
Author: camby <10...@qq.com>
AuthorDate: Wed Nov 9 16:58:08 2022 +0800
[refractor](array) refractor DataTypeArray from_string (#13905)
refractor DataTypeArray from_string, make it more clear;
support ',' and ']' inside string element, for example: ['hello,,,', 'world][]']
support empty elements, such as [,] ==> [0,0]
Co-authored-by: cambyzju <zh...@baidu.com>
---
be/src/exec/broker_scanner.cpp | 4 -
be/src/vec/data_types/data_type_array.cpp | 159 ++++++++++++---------
.../data/load_p0/broker_load/simple_array.csv | 3 +-
.../data/load_p0/broker_load/test_array_load.out | 4 +
.../data/load_p0/stream_load/test_stream_load.out | 4 +-
.../cast_function/test_cast_string_to_array.out | 9 +-
6 files changed, 108 insertions(+), 75 deletions(-)
diff --git a/be/src/exec/broker_scanner.cpp b/be/src/exec/broker_scanner.cpp
index 3d9d5cf0f2..d65b413927 100644
--- a/be/src/exec/broker_scanner.cpp
+++ b/be/src/exec/broker_scanner.cpp
@@ -495,10 +495,6 @@ Status BrokerScanner::_line_to_src_tuple(const Slice& line) {
return Status::OK();
}
- if (!check_array_format(_split_values)) {
- return Status::OK();
- }
-
for (int i = 0; i < _split_values.size(); ++i) {
auto slot_desc = _src_slot_descs[i];
const Slice& value = _split_values[i];
diff --git a/be/src/vec/data_types/data_type_array.cpp b/be/src/vec/data_types/data_type_array.cpp
index 9f29980e60..6827cb3779 100644
--- a/be/src/vec/data_types/data_type_array.cpp
+++ b/be/src/vec/data_types/data_type_array.cpp
@@ -175,6 +175,69 @@ std::string DataTypeArray::to_string(const IColumn& column, size_t row_num) cons
return ss.str();
}
+bool next_element_from_string(ReadBuffer& rb, StringRef& output, bool& has_quota) {
+ StringRef element(rb.position(), 0);
+ has_quota = false;
+ if (rb.eof()) {
+ return false;
+ }
+
+ // ltrim
+ while (!rb.eof() && isspace(*rb.position())) {
+ ++rb.position();
+ element.data = rb.position();
+ }
+
+ // parse string
+ if (*rb.position() == '"' || *rb.position() == '\'') {
+ const char str_sep = *rb.position();
+ size_t str_len = 1;
+ // search until next '"' or '\''
+ while (str_len < rb.count() && *(rb.position() + str_len) != str_sep) {
+ ++str_len;
+ }
+ // invalid string
+ if (str_len >= rb.count()) {
+ rb.position() = rb.end();
+ return false;
+ }
+ has_quota = true;
+ rb.position() += str_len + 1;
+ element.size += str_len + 1;
+ }
+
+ // parse array element until array separator ',' or end ']'
+ while (!rb.eof() && (*rb.position() != ',') && (rb.count() != 1 || *rb.position() != ']')) {
+ // invalid elements such as ["123" 456,"789" 777]
+ // correct elements such as ["123" ,"789" ]
+ if (has_quota && !isspace(*rb.position())) {
+ return false;
+ }
+ ++rb.position();
+ ++element.size;
+ }
+ // invalid array element
+ if (rb.eof()) {
+ return false;
+ }
+ // adjust read buffer position to first char of next array element
+ ++rb.position();
+
+ // rtrim
+ while (element.size > 0 && isspace(element.data[element.size - 1])) {
+ --element.size;
+ }
+
+ // trim '"' and '\'' for string
+ if (element.size >= 2 && (element.data[0] == '"' || element.data[0] == '\'') &&
+ element.data[0] == element.data[element.size - 1]) {
+ ++element.data;
+ element.size -= 2;
+ }
+ output = element;
+ return true;
+}
+
Status DataTypeArray::from_string(ReadBuffer& rb, IColumn* column) const {
DCHECK(!rb.eof());
// only support one level now
@@ -191,85 +254,53 @@ Status DataTypeArray::from_string(ReadBuffer& rb, IColumn* column) const {
return Status::InvalidArgument("Array does not end with ']' character, found '{}'",
*(rb.end() - 1));
}
+ // empty array []
+ if (rb.count() == 2) {
+ offsets.push_back(offsets.back());
+ return Status::OK();
+ }
++rb.position();
- bool first = true;
- size_t size = 0;
- while (!rb.eof() && *rb.position() != ']') {
- if (!first) {
- if (*rb.position() == ',') {
- ++rb.position();
- } else {
- return Status::InvalidArgument(
- "Cannot read array from text, expected comma or end of array, found '{}'",
- *rb.position());
- }
- }
- first = false;
- if (*rb.position() == ']') {
- break;
- }
- size_t nested_str_len = 0;
- char* temp_char = rb.position() + nested_str_len;
- while (*(temp_char) != ']' && *(temp_char) != ',' && temp_char != rb.end()) {
- ++nested_str_len;
- temp_char = rb.position() + nested_str_len;
+
+ size_t element_num = 0;
+ // parse array element until end of array
+ while (!rb.eof()) {
+ StringRef element(rb.position(), rb.count());
+ bool has_quota = false;
+ if (!next_element_from_string(rb, element, has_quota)) {
+ return Status::InvalidArgument("Cannot read array element from text '{}'",
+ element.to_string());
}
- // dispose the case of [123,,,]
- if (nested_str_len == 0) {
+ // handle empty element
+ if (element.size == 0) {
auto& nested_null_col = reinterpret_cast<ColumnNullable&>(nested_column);
nested_null_col.get_nested_column().insert_default();
nested_null_col.get_null_map_data().push_back(0);
- ++size;
+ ++element_num;
continue;
}
- // Note: here we will trim elements, such as
- // ["2020-09-01", "2021-09-01" , "2022-09-01" ] ==> ["2020-09-01","2021-09-01","2022-09-01"]
- size_t begin_pos = 0;
- size_t end_pos = nested_str_len - 1;
- while (begin_pos < end_pos) {
- if (isspace(*(rb.position() + begin_pos))) {
- ++begin_pos;
- } else if (isspace(*(rb.position() + end_pos))) {
- --end_pos;
- } else {
- break;
- }
- }
-
- // dispose the case of ["123"] or ['123']
- bool has_quota = false;
- size_t tmp_len = nested_str_len;
- ReadBuffer read_buffer(rb.position(), nested_str_len);
- auto begin_char = *(rb.position() + begin_pos);
- auto end_char = *(rb.position() + end_pos);
- if (begin_char == end_char && (begin_char == '"' || begin_char == '\'')) {
- int64_t length = end_pos - begin_pos - 1;
- read_buffer = ReadBuffer(rb.position() + begin_pos + 1, (length > 0 ? length : 0));
- tmp_len = (length > 0 ? length : 0);
- has_quota = true;
- }
-
- // handle null, need to distinguish null and "null"
- if (!has_quota && tmp_len == 4 && strncmp(read_buffer.position(), "null", 4) == 0) {
+ // handle null element, need to distinguish null and "null"
+ if (!has_quota && element.size == 4 && strncmp(element.data, "null", 4) == 0) {
// insert null
auto& nested_null_col = reinterpret_cast<ColumnNullable&>(nested_column);
nested_null_col.get_nested_column().insert_default();
nested_null_col.get_null_map_data().push_back(1);
- } else {
- auto st = nested->from_string(read_buffer, &nested_column);
- if (!st.ok()) {
- // we should do revert if error
- array_column->pop_back(size);
- return st;
- }
+ ++element_num;
+ continue;
+ }
+
+ // handle normal element
+ ReadBuffer read_buffer(const_cast<char*>(element.data), element.size);
+ auto st = nested->from_string(read_buffer, &nested_column);
+ if (!st.ok()) {
+ // we should do revert if error
+ array_column->pop_back(element_num);
+ return st;
}
- rb.position() += nested_str_len;
- DCHECK_LE(rb.position(), rb.end());
- ++size;
+ ++element_num;
}
- offsets.push_back(offsets.back() + size);
+ offsets.push_back(offsets.back() + element_num);
return Status::OK();
}
diff --git a/regression-test/data/load_p0/broker_load/simple_array.csv b/regression-test/data/load_p0/broker_load/simple_array.csv
index 0514c702de..3dbe488757 100644
--- a/regression-test/data/load_p0/broker_load/simple_array.csv
+++ b/regression-test/data/load_p0/broker_load/simple_array.csv
@@ -2,4 +2,5 @@
2/[6,7,8,9,10]/[32767,32768,32769]/[65534,65535,65536]/["a","b","c","d","e"]/["hello","world"]/["1991-01-01"]/["1991-01-01 00:00:00"]/[0.33,0.67]/[3.1415926,0.878787878]/[1,1.2,1.3]
3/[]/[32767,32768,32769]/[null,null,65536]/["a","b","c","d","e"]/["happy","birthday"]/["1991-01-01"]/["1991-01-01 00:00:00"]/[0.33,0.67]/[3.1415926,0.878787878]/[1,1.2,1.3]
4/[null]/[32767,32768,32769]/[ null,null,65536]/["a","b","c","d","e"]/["hello","world"]/["1991-01-01"]/["1991-01-01 00:00:00"]/[0.33,0.67]/[3.1415926,0.878787878]/[1,1.2,1.3]
-5/[null,null]/[32767,32768,null]/[65534,null,65536]/["a","b","c","d","e"]/["hello","world"]/["1991-01-01"]/["1991-01-01 00:00:00"]/[0.33,0.67]/[3.1415926,0.878787878]/[1,1.2,1.3]
\ No newline at end of file
+5/[null,null]/[32767,32768,null]/[65534,null,65536]/["a","b","c","d","e"]/["hello","world"]/["1991-01-01"]/["1991-01-01 00:00:00"]/[0.33,0.67]/[3.1415926,0.878787878]/[1,1.2,1.3]
+6/[null,null,,]/[ 32767, 32768 ,NULL]/[65534,null,65536 ]/["a","b","c","d","e"]/["hello,,,","world][]"]/[ "1991-01-01" ]/["1991-01-01 00:00:00"]/[0.33,0.67]/[3.1415926,0.878787878]/[,]
\ No newline at end of file
diff --git a/regression-test/data/load_p0/broker_load/test_array_load.out b/regression-test/data/load_p0/broker_load/test_array_load.out
index 6c568b6ef1..ff4d69cbff 100644
--- a/regression-test/data/load_p0/broker_load/test_array_load.out
+++ b/regression-test/data/load_p0/broker_load/test_array_load.out
@@ -21,6 +21,7 @@
3 [] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['happy', 'birthday'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1, 1.2, 1.3]
4 [NULL] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1, 1.2, 1.3]
5 [NULL, NULL] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1, 1.2, 1.3]
+6 [NULL, NULL, 0, 0] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello,,,', 'world][]'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [0, 0]
100 [1, 2, 3] [32767, 32768, 32769] [65534, 65535, 65536] ['a', 'b', 'c'] ['hello', 'world'] [2022-07-13] [2022-07-13 12:30:00] [0.33, 0.67] [3.1415926, 0.878787878] [4, 5.5, 6.67]
-- !select --
@@ -29,6 +30,7 @@
3 [] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['happy', 'birthday'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1.000000, 1.200000, 1.300000]
4 [NULL] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1.000000, 1.200000, 1.300000]
5 [NULL, NULL] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1.000000, 1.200000, 1.300000]
+6 [NULL, NULL, 0, 0] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello,,,', 'world][]'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [0.000000, 0.000000]
100 [1, 2, 3] [32767, 32768, 32769] [65534, 65535, 65536] ['a', 'b', 'c'] ['hello', 'world'] [2022-07-13] [2022-07-13 12:30:00] [0.33, 0.67] [3.1415926, 0.878787878] [4.000000, 5.500000, 6.670000]
-- !select --
@@ -61,6 +63,7 @@
3 [] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['happy', 'birthday'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1, 1.2, 1.3]
4 [NULL] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1, 1.2, 1.3]
5 [NULL, NULL] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1, 1.2, 1.3]
+6 [NULL, NULL, 0, 0] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello,,,', 'world][]'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [0, 0]
100 [1, 2, 3] [32767, 32768, 32769] [65534, 65535, 65536] ['a', 'b', 'c'] ['hello', 'world'] [2022-07-13] [2022-07-13 12:30:00] [0.33, 0.67] [3.1415926, 0.878787878] [4, 5.5, 6.67]
-- !select --
@@ -69,6 +72,7 @@
3 [] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['happy', 'birthday'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1.000000, 1.200000, 1.300000]
4 [NULL] [32767, 32768, 32769] [NULL, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1.000000, 1.200000, 1.300000]
5 [NULL, NULL] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello', 'world'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [1.000000, 1.200000, 1.300000]
+6 [NULL, NULL, 0, 0] [32767, 32768, NULL] [65534, NULL, 65536] ['a', 'b', 'c', 'd', 'e'] ['hello,,,', 'world][]'] [1991-01-01] [1991-01-01 00:00:00] [0.33, 0.67] [3.1415926, 0.878787878] [0.000000, 0.000000]
100 [1, 2, 3] [32767, 32768, 32769] [65534, 65535, 65536] ['a', 'b', 'c'] ['hello', 'world'] [2022-07-13] [2022-07-13 12:30:00] [0.33, 0.67] [3.1415926, 0.878787878] [4.000000, 5.500000, 6.670000]
-- !select --
diff --git a/regression-test/data/load_p0/stream_load/test_stream_load.out b/regression-test/data/load_p0/stream_load/test_stream_load.out
index 162f76ca74..85a8835f51 100644
--- a/regression-test/data/load_p0/stream_load/test_stream_load.out
+++ b/regression-test/data/load_p0/stream_load/test_stream_load.out
@@ -4,7 +4,7 @@
-2 -50 \N 1 \N \N \N \N \N \N \N \N j \N
-- !sql1 --
-2019 9 9 9 7.7 a 2019-09-09 1970-01-01T08:33:39 k7 9.0 9.0
+2019 9 9 9 7.700 a 2019-09-09 1970-01-01T08:33:39 k7 9.0 9.0
-- !all11 --
2500
@@ -67,5 +67,5 @@
6 [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, 'null'] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL, NULL, NULL, NULL, NULL]
6 [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL] [NULL, NULL, NULL, NULL, NULL, NULL]
7 [1, 2, 3, 4, 5] \N \N \N \N \N \N \N \N \N
-8 [1, 2, 3, 4, 5] \N \N \N \N \N [] \N [NULL] \N
+8 [1, 2, 3, 4, 5] \N \N \N \N \N [NULL] \N [NULL] \N
diff --git a/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out b/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
index 1eb1418926..90d372c356 100644
--- a/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
+++ b/regression-test/data/query_p0/sql_functions/cast_function/test_cast_string_to_array.out
@@ -15,13 +15,14 @@
[2022-09-01]
-- !sql --
-[1, 2, 3, 0, 0]
+[1, 2, 3, 0, 0, 0]
-- !sql --
-['a', 'b', 'c', '', '']
+['a', 'b', 'c', '', '', '']
-- !sql --
-[1.34, 2.01, 0, 0]
+[1.34, 2.01, 0, 0, 0]
-- !sql --
-[2022-09-01, 0000-00-00]
+[2022-09-01, 0000-00-00, 0000-00-00]
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 26/39: [fix](ctas) text column type len = 1 when create table as select (#13906)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ddb57a4eebe9ed4166c82ac926e74b6d34af5b44
Author: xueweizhang <zx...@163.com>
AuthorDate: Wed Nov 9 09:09:34 2022 +0800
[fix](ctas) text column type len = 1 when create table as select (#13906)
Signed-off-by: nextdreamblue <zx...@163.com>
---
.../java/org/apache/doris/analysis/ColumnDef.java | 7 ++++++-
.../org/apache/doris/datasource/InternalCatalog.java | 3 ++-
regression-test/data/ddl_p0/test_ctas.out | 8 +++++++-
regression-test/suites/ddl_p0/test_ctas.groovy | 19 ++++++++++++++++++-
4 files changed, 33 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
index d6b7981993..170ceaea4c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
@@ -206,7 +206,12 @@ public class ColumnDef {
if (typeDef.getType().isScalarType()) {
final ScalarType targetType = (ScalarType) typeDef.getType();
if (targetType.getPrimitiveType().isStringType() && !targetType.isLengthSet()) {
- targetType.setLength(1);
+ if (targetType.getPrimitiveType() != PrimitiveType.STRING) {
+ targetType.setLength(1);
+ } else {
+ // alway set text length MAX_STRING_LENGTH
+ targetType.setLength(ScalarType.MAX_STRING_LENGTH);
+ }
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index c66ee75e46..6bfec923e2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -1206,7 +1206,8 @@ public class InternalCatalog implements CatalogIf<Database> {
Expr resultExpr = resultExprs.get(i);
Type resultType = resultExpr.getType();
if (resultType.isStringType() && resultType.getLength() < 0) {
- typeDef = new TypeDef(Type.STRING);
+ // alway set text length MAX_STRING_LENGTH
+ typeDef = new TypeDef(ScalarType.createStringType());
} else if (resultType.isDecimalV2() && resultType.equals(ScalarType.DECIMALV2)) {
typeDef = new TypeDef(ScalarType.createDecimalType(27, 9));
} else if (resultType.isDecimalV3()) {
diff --git a/regression-test/data/ddl_p0/test_ctas.out b/regression-test/data/ddl_p0/test_ctas.out
index aa41c6a791..2a8f9f7979 100644
--- a/regression-test/data/ddl_p0/test_ctas.out
+++ b/regression-test/data/ddl_p0/test_ctas.out
@@ -1,6 +1,12 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !select --
-test_ctas1 CREATE TABLE `test_ctas1` (\n `test_varchar` varchar(150) NULL,\n `test_datetime` datetime NULL,\n `test_default_timestamp` datetime NULL DEFAULT CURRENT_TIMESTAMP\n) ENGINE=OLAP\nDUPLICATE KEY(`test_varchar`)\nCOMMENT 'OLAP'\nDISTRIBUTED BY HASH(`test_varchar`) BUCKETS 10\nPROPERTIES (\n"replication_allocation" = "tag.location.default: 1",\n"in_memory" = "false",\n"storage_format" = "V2",\n"disable_auto_compaction" = "false"\n);
+test_ctas1 CREATE TABLE `test_ctas1` (\n `test_varchar` varchar(150) NULL,\n `test_text` text NULL,\n `test_datetime` datetime NULL,\n `test_default_timestamp` datetime NULL DEFAULT CURRENT_TIMESTAMP\n) ENGINE=OLAP\nDUPLICATE KEY(`test_varchar`)\nCOMMENT 'OLAP'\nDISTRIBUTED BY HASH(`test_varchar`) BUCKETS 10\nPROPERTIES (\n"replication_allocation" = "tag.location.default: 1",\n"in_memory" = "false",\n"storage_format" = "V2",\n"disable_auto_compaction" = "false"\n);
+
+-- !select --
+2
+
+-- !select --
+test_ctas2 CREATE TABLE `test_ctas2` (\n `test_varchar` varchar(150) NULL,\n `test_text` text NULL,\n `test_datetime` datetime NULL,\n `test_default_timestamp` datetime NULL DEFAULT CURRENT_TIMESTAMP\n) ENGINE=OLAP\nDUPLICATE KEY(`test_varchar`)\nCOMMENT 'OLAP'\nDISTRIBUTED BY HASH(`test_varchar`) BUCKETS 10\nPROPERTIES (\n"replication_allocation" = "tag.location.default: 1",\n"in_memory" = "false",\n"storage_format" = "V2",\n"disable_auto_compaction" = "false"\n);
-- !select --
2
diff --git a/regression-test/suites/ddl_p0/test_ctas.groovy b/regression-test/suites/ddl_p0/test_ctas.groovy
index 74403f7602..2a94e82b0e 100644
--- a/regression-test/suites/ddl_p0/test_ctas.groovy
+++ b/regression-test/suites/ddl_p0/test_ctas.groovy
@@ -20,6 +20,7 @@ suite("test_ctas") {
sql """
CREATE TABLE IF NOT EXISTS `test_ctas` (
`test_varchar` varchar(150) NULL,
+ `test_text` text NULL,
`test_datetime` datetime NULL,
`test_default_timestamp` datetime DEFAULT CURRENT_TIMESTAMP
) ENGINE=OLAP
@@ -32,7 +33,7 @@ suite("test_ctas") {
)
"""
- sql """ INSERT INTO test_ctas(test_varchar, test_datetime) VALUES ('test1','2022-04-27 16:00:33'),('test2','2022-04-27 16:00:54') """
+ sql """ INSERT INTO test_ctas(test_varchar, test_text, test_datetime) VALUES ('test1','test11','2022-04-27 16:00:33'),('test2','test22','2022-04-27 16:00:54') """
sql """
CREATE TABLE IF NOT EXISTS `test_ctas1`
@@ -46,10 +47,26 @@ suite("test_ctas") {
qt_select """SHOW CREATE TABLE `test_ctas1`"""
qt_select """select count(*) from test_ctas1"""
+
+ sql """
+ CREATE TABLE IF NOT EXISTS `test_ctas2`
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "in_memory" = "false",
+ "storage_format" = "V2"
+ ) as select test_varchar, lpad(test_text,10,'0') as test_text, test_datetime, test_default_timestamp from test_ctas;
+ """
+
+ qt_select """SHOW CREATE TABLE `test_ctas2`"""
+
+ qt_select """select count(*) from test_ctas2"""
} finally {
sql """ DROP TABLE IF EXISTS test_ctas """
sql """ DROP TABLE IF EXISTS test_ctas1 """
+
+ sql """ DROP TABLE IF EXISTS test_ctas2 """
}
}
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 39/39: [Enhancement][fix](profile)() modify some profiles (#14074)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3fff8f399baad5f060a1992447a854c165f203f2
Author: Pxl <px...@qq.com>
AuthorDate: Wed Nov 9 21:59:28 2022 +0800
[Enhancement][fix](profile)() modify some profiles (#14074)
1. add RemainedDownPredicates
2. fix core dump when _scan_ranges is empty
3. fix invalid memory access on vLiteral's debug_string()
4. enlarge mv test wait time
---
be/src/vec/exec/scan/new_olap_scan_node.cpp | 9 +++++++++
be/src/vec/exprs/vliteral.cpp | 2 ++
regression-test/suites/rollup/test_materialized_view_date.groovy | 8 ++++----
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/be/src/vec/exec/scan/new_olap_scan_node.cpp b/be/src/vec/exec/scan/new_olap_scan_node.cpp
index a1a52a72ff..1520c834f4 100644
--- a/be/src/vec/exec/scan/new_olap_scan_node.cpp
+++ b/be/src/vec/exec/scan/new_olap_scan_node.cpp
@@ -134,6 +134,9 @@ static std::string olap_filters_to_string(const std::vector<doris::TCondition>&
static std::string tablets_id_to_string(
const std::vector<std::unique_ptr<TPaloScanRange>>& scan_ranges) {
+ if (scan_ranges.empty()) {
+ return "[empty]";
+ }
std::stringstream ss;
ss << "[" << scan_ranges[0]->tablet_id;
for (int i = 1; i < scan_ranges.size(); ++i) {
@@ -279,6 +282,12 @@ Status NewOlapScanNode::_init_scanners(std::list<VScanner*>* scanners) {
_eos = true;
return Status::OK();
}
+
+ if (_vconjunct_ctx_ptr && (*_vconjunct_ctx_ptr)->root()) {
+ _runtime_profile->add_info_string("RemainedDownPredicates",
+ (*_vconjunct_ctx_ptr)->root()->debug_string());
+ }
+
auto span = opentelemetry::trace::Tracer::GetCurrentSpan();
// ranges constructed from scan keys
diff --git a/be/src/vec/exprs/vliteral.cpp b/be/src/vec/exprs/vliteral.cpp
index 680023010f..4f9ad2e7a2 100644
--- a/be/src/vec/exprs/vliteral.cpp
+++ b/be/src/vec/exprs/vliteral.cpp
@@ -205,7 +205,9 @@ std::string VLiteral::debug_string() const {
switch (_type.type) {
case TYPE_BOOLEAN:
case TYPE_TINYINT:
+ out << *(reinterpret_cast<const int8_t*>(ref.data));
case TYPE_SMALLINT:
+ out << *(reinterpret_cast<const int16_t*>(ref.data));
case TYPE_INT: {
out << *(reinterpret_cast<const int32_t*>(ref.data));
break;
diff --git a/regression-test/suites/rollup/test_materialized_view_date.groovy b/regression-test/suites/rollup/test_materialized_view_date.groovy
index e927e3c87c..ce69400f6c 100644
--- a/regression-test/suites/rollup/test_materialized_view_date.groovy
+++ b/regression-test/suites/rollup/test_materialized_view_date.groovy
@@ -37,7 +37,7 @@ suite("test_materialized_view_date", "rollup") {
DISTRIBUTED BY HASH(record_id) properties("replication_num" = "1");
"""
- int max_try_secs = 60
+ int max_try_secs = 120
sql "CREATE materialized VIEW amt_max1 AS SELECT store_id, max(sale_date1) FROM ${tbName1} GROUP BY store_id;"
while (max_try_secs--) {
String res = getJobState(tbName1)
@@ -52,7 +52,7 @@ suite("test_materialized_view_date", "rollup") {
}
}
Thread.sleep(2)
- max_try_secs = 60
+ max_try_secs = 120
sql "CREATE materialized VIEW amt_max2 AS SELECT store_id, max(sale_datetime1) FROM ${tbName1} GROUP BY store_id;"
while (max_try_secs--) {
String res = getJobState(tbName1)
@@ -67,7 +67,7 @@ suite("test_materialized_view_date", "rollup") {
}
}
Thread.sleep(2)
- max_try_secs = 60
+ max_try_secs = 120
sql "CREATE materialized VIEW amt_max3 AS SELECT store_id, max(sale_datetime2) FROM ${tbName1} GROUP BY store_id;"
while (max_try_secs--) {
String res = getJobState(tbName1)
@@ -82,7 +82,7 @@ suite("test_materialized_view_date", "rollup") {
}
}
Thread.sleep(2)
- max_try_secs = 60
+ max_try_secs = 120
sql "CREATE materialized VIEW amt_max4 AS SELECT store_id, max(sale_datetime3) FROM ${tbName1} GROUP BY store_id;"
while (max_try_secs--) {
String res = getJobState(tbName1)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 05/39: [feature-wip](multi-catalog) fix page index filter bug (#14015)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2d8d8233b6faa69200c2a366058d952409f3d4d4
Author: slothever <18...@users.noreply.github.com>
AuthorDate: Tue Nov 8 12:10:12 2022 +0800
[feature-wip](multi-catalog) fix page index filter bug (#14015)
Fix page index filter not take effect when multiple columns
Co-authored-by: jinzhe <ji...@selectdb.com>
---
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/vec/exec/format/parquet/vparquet_reader.cpp b/be/src/vec/exec/format/parquet/vparquet_reader.cpp
index dc33f321d7..af899fb318 100644
--- a/be/src/vec/exec/format/parquet/vparquet_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_reader.cpp
@@ -337,7 +337,7 @@ Status ParquetReader::_process_page_index(const tparquet::RowGroup& row_group,
page_index.collect_skipped_page_range(&column_index, conjuncts, col_schema,
skipped_page_range);
if (skipped_page_range.empty()) {
- return Status::OK();
+ break;
}
tparquet::OffsetIndex offset_index;
RETURN_IF_ERROR(page_index.parse_offset_index(chunk, buff, buffer_size, &offset_index));
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 17/39: use extern template to date_time_add (#13970)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f56fe5ff3597aee7a06cea3e2b8eb37d98bda93c
Author: Pxl <px...@qq.com>
AuthorDate: Tue Nov 8 22:11:41 2022 +0800
use extern template to date_time_add (#13970)
---
be/src/vec/functions/function_date_or_datetime_computation.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/vec/functions/function_date_or_datetime_computation.h b/be/src/vec/functions/function_date_or_datetime_computation.h
index 4d1002ed0e..a22bc1a357 100644
--- a/be/src/vec/functions/function_date_or_datetime_computation.h
+++ b/be/src/vec/functions/function_date_or_datetime_computation.h
@@ -34,7 +34,7 @@ namespace doris::vectorized {
template <TimeUnit unit, typename Arg, typename DateValueType, typename ResultDateValueType,
typename ResultType>
-inline ResultType date_time_add(const Arg& t, Int64 delta, bool& is_null) {
+extern ResultType date_time_add(const Arg& t, Int64 delta, bool& is_null) {
auto ts_value = binary_cast<Arg, DateValueType>(t);
TimeInterval interval(unit, delta, false);
if constexpr (std::is_same_v<VecDateTimeValue, DateValueType> ||
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 16/39: [fix](keyranges) fix the split error of keyranges (#14049)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 087f168f2f9bef65b16cc6af492d43f4a58dbf04
Author: luozenglin <37...@users.noreply.github.com>
AuthorDate: Tue Nov 8 22:09:16 2022 +0800
[fix](keyranges) fix the split error of keyranges (#14049)
fix the split error of keyranges
---
be/src/exec/olap_common.h | 28 +++++++++++++---------
be/test/exec/olap_common_test.cpp | 2 +-
.../data/query_p0/aggregate/aggregate.out | 3 +++
.../suites/query_p0/aggregate/aggregate.groovy | 2 ++
4 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/be/src/exec/olap_common.h b/be/src/exec/olap_common.h
index aaa76f9246..8c470316bb 100644
--- a/be/src/exec/olap_common.h
+++ b/be/src/exec/olap_common.h
@@ -111,8 +111,8 @@ public:
void convert_to_range_value();
bool convert_to_avg_range_value(std::vector<OlapTuple>& begin_scan_keys,
- std::vector<OlapTuple>& end_scan_keys,
- int32_t max_scan_key_num);
+ std::vector<OlapTuple>& end_scan_keys, bool& begin_include,
+ bool& end_include, int32_t max_scan_key_num);
bool has_intersection(ColumnValueRange<primitive_type>& range);
@@ -516,7 +516,7 @@ size_t ColumnValueRange<primitive_type>::get_convertible_fixed_value_size() cons
template <PrimitiveType primitive_type>
bool ColumnValueRange<primitive_type>::convert_to_avg_range_value(
std::vector<OlapTuple>& begin_scan_keys, std::vector<OlapTuple>& end_scan_keys,
- int32_t max_scan_key_num) {
+ bool& begin_include, bool& end_include, int32_t max_scan_key_num) {
constexpr bool reject_type = primitive_type == PrimitiveType::TYPE_LARGEINT ||
primitive_type == PrimitiveType::TYPE_DECIMALV2 ||
primitive_type == PrimitiveType::TYPE_HLL ||
@@ -526,6 +526,8 @@ bool ColumnValueRange<primitive_type>::convert_to_avg_range_value(
primitive_type == PrimitiveType::TYPE_BOOLEAN ||
primitive_type == PrimitiveType::TYPE_DATETIME ||
primitive_type == PrimitiveType::TYPE_DATETIMEV2;
+ begin_include = is_begin_include();
+ end_include = is_end_include();
if constexpr (reject_type) {
begin_scan_keys.emplace_back();
begin_scan_keys.back().add_value(
@@ -537,8 +539,14 @@ bool ColumnValueRange<primitive_type>::convert_to_avg_range_value(
return true;
} else {
CppType current = get_range_min_value();
+ CppType max_value = get_range_max_value();
+ if (!is_begin_include() && !is_end_include() && current < TYPE_MAX &&
+ current + 1 < max_value) {
+ begin_include = true;
+ ++current;
+ }
- size_t range_size = get_convertible_fixed_value_size();
+ size_t range_size = is_fixed_value_convertible() ? max_value - current : 0;
size_t step_size = std::max(
(range_size / max_scan_key_num) + (range_size % max_scan_key_num != 0), (size_t)1);
@@ -546,13 +554,13 @@ bool ColumnValueRange<primitive_type>::convert_to_avg_range_value(
current.set_type(TimeType::TIME_DATE);
}
- while (current < get_range_max_value()) {
+ while (current < max_value) {
begin_scan_keys.emplace_back();
begin_scan_keys.back().add_value(
cast_to_string<primitive_type, CppType>(current, scale()));
- if (get_range_max_value() - current < step_size) {
- current = get_range_max_value();
+ if (max_value - current < step_size) {
+ current = max_value;
} else {
current += step_size;
}
@@ -893,12 +901,10 @@ Status OlapScanKeys::extend_scan_key(ColumnValueRange<primitive_type>& range,
}
} else {
if (_begin_scan_keys.empty() && range.is_fixed_value_convertible() && _is_convertible) {
- if (range.convert_to_avg_range_value(_begin_scan_keys, _end_scan_keys,
- max_scan_key_num)) {
+ if (range.convert_to_avg_range_value(_begin_scan_keys, _end_scan_keys, _begin_include,
+ _end_include, max_scan_key_num)) {
_has_range_value = true;
}
- _begin_include = range.is_begin_include();
- _end_include = range.is_end_include();
return Status::OK();
}
}
diff --git a/be/test/exec/olap_common_test.cpp b/be/test/exec/olap_common_test.cpp
index 9e77b3d455..e0056cad41 100644
--- a/be/test/exec/olap_common_test.cpp
+++ b/be/test/exec/olap_common_test.cpp
@@ -671,7 +671,7 @@ TEST_F(OlapScanKeysTest, EachtypeTest) {
scan_keys.get_key_range(&key_range);
EXPECT_EQ(key_range.size(), max_scan_key);
- EXPECT_EQ(OlapScanKeys::to_print_key(key_range[0]->begin_scan_range), "0");
+ EXPECT_EQ(OlapScanKeys::to_print_key(key_range[0]->begin_scan_range), "1");
EXPECT_EQ(OlapScanKeys::to_print_key(key_range[max_scan_key - 1]->end_scan_range), "32766");
}
}
diff --git a/regression-test/data/query_p0/aggregate/aggregate.out b/regression-test/data/query_p0/aggregate/aggregate.out
index f571fd92a5..992e4cc4da 100644
--- a/regression-test/data/query_p0/aggregate/aggregate.out
+++ b/regression-test/data/query_p0/aggregate/aggregate.out
@@ -660,3 +660,6 @@ TESTING AGAIN
14 1 11011902 1
15 1 11011920 1
+-- !aggregate31 --
+15
+
diff --git a/regression-test/suites/query_p0/aggregate/aggregate.groovy b/regression-test/suites/query_p0/aggregate/aggregate.groovy
index 513b98b34a..76abb4af0a 100644
--- a/regression-test/suites/query_p0/aggregate/aggregate.groovy
+++ b/regression-test/suites/query_p0/aggregate/aggregate.groovy
@@ -283,4 +283,6 @@ suite("aggregate") {
qt_aggregate_2phase_0"""select avg(distinct k1),avg(k2) from baseall"""
qt_aggregate_2phase_1"""select k1,count(distinct k2,k3),min(k4),count(*) from baseall group by k1 order by k1"""
+
+ qt_aggregate31"select count(*) from baseall where k1 < 64 and k1 > 0;"
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 06/39: [fix](Nereids) Use simple cost to calculate benefit and avoid unuseless calculation (#14056)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1979312b4768b6ca080128f5a7cf844aee000574
Author: 谢健 <ji...@gmail.com>
AuthorDate: Tue Nov 8 13:11:38 2022 +0800
[fix](Nereids) Use simple cost to calculate benefit and avoid unuseless calculation (#14056)
In GraphSimplifier, we can use simple cost to calculate the benefit.
And only when the best neighbor of the apply step is the processing edge, we need to update recursively.
---
.../joinreorder/hypergraph/GraphSimplifier.java | 11 +++++------
.../nereids/rules/joinreorder/hypergraph/Node.java | 1 -
.../joinreorder/hypergraph/GraphSimplifierTest.java | 20 ++++++++++----------
3 files changed, 15 insertions(+), 17 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifier.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifier.java
index 693b0f2044..1bfb0e1a38 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifier.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifier.java
@@ -128,7 +128,9 @@ public class GraphSimplifier {
edgeIndex1)) {
continue;
}
- processNeighbors(edgeIndex2, 0, edgeSize);
+ if (bestSimplification.bestNeighbor == edgeIndex1) {
+ processNeighbors(edgeIndex2, 0, edgeSize);
+ }
}
// Go through the neighbors with higher index, we only need to recalculate all related steps
@@ -289,17 +291,14 @@ public class GraphSimplifier {
private LogicalJoin simulateJoin(Plan plan1, LogicalJoin join, Plan plan2) {
// In Graph Simplifier, we use the simple cost model, that is
- // Plan.cost = Plan.rowCount + Plan.children1.cost + ...
+ // Plan.cost = Plan.rowCount + Plan.children1.cost + Plan.children2.cost
// The reason is that this cost model has ASI (adjacent sequence interchange) property.
// TODO: consider network, data distribution cost
- double cost = 0;
LogicalJoin newJoin = new LogicalJoin(join.getJoinType(), plan1, plan2);
List<Group> children = new ArrayList<>();
- cost += getGroup(plan1).getStatistics().getRowCount();
children.add(getGroup(plan1));
- cost += getGroup(plan2).getStatistics().getRowCount();
children.add(getGroup(plan2));
-
+ double cost = getSimpleCost(plan1) + getSimpleCost(plan2);
GroupExpression groupExpression = new GroupExpression(newJoin, children);
Group group = new Group();
group.addGroupExpression(groupExpression);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/Node.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/Node.java
index 8fa93d223f..a2d7696c85 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/Node.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/Node.java
@@ -97,7 +97,6 @@ class Node {
} else {
complexEdges.remove(edge);
}
- assert false : String.format("removed edge %d at %d that does not exists", edge.getIndex(), index);
}
public String getName() {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifierTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifierTest.java
index d0c71ebfd2..6f599ef763 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifierTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/joinreorder/hypergraph/GraphSimplifierTest.java
@@ -48,22 +48,22 @@ public class GraphSimplifierTest {
+ " rowCount=40.00\"];\n"
+ " LOGICAL_OLAP_SCAN4 [label=\"LOGICAL_OLAP_SCAN4 \n"
+ " rowCount=50.00\"];\n"
- + "e0 [shape=circle, width=.001, label=\"\"]\n"
- + "LOGICAL_OLAP_SCAN0 -> e0 [arrowhead=none, label=\"0.10\"]\n"
- + "LOGICAL_OLAP_SCAN2 -> e0 [arrowhead=none, label=\"0.10\"]\n"
- + "LOGICAL_OLAP_SCAN3 -> e0 [arrowhead=none, label=\"0.10\"]\n"
- + "LOGICAL_OLAP_SCAN4 -> e0 [arrowhead=none, label=\"0.10\"]\n"
- + "LOGICAL_OLAP_SCAN1 -> e0 [arrowhead=none, label=\"\"]\n"
+ + "LOGICAL_OLAP_SCAN0 -> LOGICAL_OLAP_SCAN1 [label=\"0.10\",arrowhead=none]\n"
+ "e1 [shape=circle, width=.001, label=\"\"]\n"
+ "LOGICAL_OLAP_SCAN0 -> e1 [arrowhead=none, label=\"0.05\"]\n"
- + "LOGICAL_OLAP_SCAN3 -> e1 [arrowhead=none, label=\"0.05\"]\n"
- + "LOGICAL_OLAP_SCAN4 -> e1 [arrowhead=none, label=\"0.05\"]\n"
+ + "LOGICAL_OLAP_SCAN1 -> e1 [arrowhead=none, label=\"0.05\"]\n"
+ "LOGICAL_OLAP_SCAN2 -> e1 [arrowhead=none, label=\"\"]\n"
+ "e2 [shape=circle, width=.001, label=\"\"]\n"
+ "LOGICAL_OLAP_SCAN0 -> e2 [arrowhead=none, label=\"0.03\"]\n"
- + "LOGICAL_OLAP_SCAN4 -> e2 [arrowhead=none, label=\"0.03\"]\n"
+ + "LOGICAL_OLAP_SCAN1 -> e2 [arrowhead=none, label=\"0.03\"]\n"
+ + "LOGICAL_OLAP_SCAN2 -> e2 [arrowhead=none, label=\"0.03\"]\n"
+ "LOGICAL_OLAP_SCAN3 -> e2 [arrowhead=none, label=\"\"]\n"
- + "LOGICAL_OLAP_SCAN0 -> LOGICAL_OLAP_SCAN4 [label=\"0.03\",arrowhead=none]\n"
+ + "e3 [shape=circle, width=.001, label=\"\"]\n"
+ + "LOGICAL_OLAP_SCAN0 -> e3 [arrowhead=none, label=\"0.03\"]\n"
+ + "LOGICAL_OLAP_SCAN1 -> e3 [arrowhead=none, label=\"0.03\"]\n"
+ + "LOGICAL_OLAP_SCAN2 -> e3 [arrowhead=none, label=\"0.03\"]\n"
+ + "LOGICAL_OLAP_SCAN3 -> e3 [arrowhead=none, label=\"0.03\"]\n"
+ + "LOGICAL_OLAP_SCAN4 -> e3 [arrowhead=none, label=\"\"]\n"
+ "}\n";
String dottyGraph = hyperGraph.toDottyHyperGraph();
assert dottyGraph.equals(target) : dottyGraph;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 21/39: [refactor](new-scan) remove old vectorized scan node (#14029)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f4d2c9aaabc59c795a3e1cc70b1f8d4af89fad05
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Wed Nov 9 08:39:20 2022 +0800
[refactor](new-scan) remove old vectorized scan node (#14029)
---
be/src/common/config.h | 4 -
be/src/exec/es_http_scan_node.cpp | 15 +-
be/src/exec/es_http_scan_node.h | 6 -
be/src/exec/exec_node.cpp | 37 +-
be/src/exec/olap_scan_node.h | 1 -
be/src/exprs/expr_context.h | 5 -
be/src/vec/CMakeLists.txt | 6 -
be/src/vec/exec/ves_http_scan_node.cpp | 447 --------
be/src/vec/exec/ves_http_scan_node.h | 113 --
be/src/vec/exec/ves_http_scanner.cpp | 52 -
be/src/vec/exec/ves_http_scanner.h | 38 -
be/src/vec/exec/vjdbc_connector.cpp | 2 +-
be/src/vec/exec/vjdbc_connector.h | 2 +-
be/src/vec/exec/vjdbc_scan_node.cpp | 173 ---
be/src/vec/exec/vjdbc_scan_node.h | 68 --
be/src/vec/exec/vodbc_scan_node.cpp | 249 -----
be/src/vec/exec/vodbc_scan_node.h | 87 --
be/src/vec/exec/volap_scan_node.cpp | 1843 --------------------------------
be/src/vec/exec/volap_scan_node.h | 346 ------
be/src/vec/exec/volap_scanner.cpp | 477 ---------
be/src/vec/exec/volap_scanner.h | 153 ---
21 files changed, 13 insertions(+), 4111 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 6843a88514..bc3c67d92c 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -822,10 +822,6 @@ CONF_Int32(doris_remote_scanner_thread_pool_thread_num, "16");
// number of s3 scanner thread pool queue size
CONF_Int32(doris_remote_scanner_thread_pool_queue_size, "10240");
-// If set to true, the new scan node framework will be used.
-// This config should be removed when the new scan node is ready.
-CONF_Bool(enable_new_scan_node, "true");
-
// limit the queue of pending batches which will be sent by a single nodechannel
CONF_mInt64(nodechannel_pending_queue_max_bytes, "67108864");
diff --git a/be/src/exec/es_http_scan_node.cpp b/be/src/exec/es_http_scan_node.cpp
index 38de757f87..cea7068631 100644
--- a/be/src/exec/es_http_scan_node.cpp
+++ b/be/src/exec/es_http_scan_node.cpp
@@ -456,17 +456,10 @@ void EsHttpScanNode::scanner_worker(int start_idx, int length, std::promise<Stat
properties, _column_names, _predicates, _docvalue_context, &doc_value_mode);
// start scanner to scan
- if (!_vectorized) {
- std::unique_ptr<EsHttpScanner> scanner(
- new EsHttpScanner(_runtime_state, runtime_profile(), _tuple_id, properties,
- scanner_expr_ctxs, &counter, doc_value_mode));
- status = scanner_scan(std::move(scanner), scanner_expr_ctxs, &counter);
- } else {
- std::unique_ptr<vectorized::VEsHttpScanner> scanner(new vectorized::VEsHttpScanner(
- _runtime_state, runtime_profile(), _tuple_id, properties, scanner_expr_ctxs,
- &counter, doc_value_mode));
- status = scanner_scan(std::move(scanner));
- }
+ std::unique_ptr<EsHttpScanner> scanner(
+ new EsHttpScanner(_runtime_state, runtime_profile(), _tuple_id, properties,
+ scanner_expr_ctxs, &counter, doc_value_mode));
+ status = scanner_scan(std::move(scanner), scanner_expr_ctxs, &counter);
if (!status.ok()) {
LOG(WARNING) << "Scanner[" << start_idx
<< "] process failed. status=" << status.get_error_msg();
diff --git a/be/src/exec/es_http_scan_node.h b/be/src/exec/es_http_scan_node.h
index 0fc9fac376..d07f68af24 100644
--- a/be/src/exec/es_http_scan_node.h
+++ b/be/src/exec/es_http_scan_node.h
@@ -30,7 +30,6 @@
#include "exec/es_http_scanner.h"
#include "exec/scan_node.h"
#include "gen_cpp/PaloInternalService_types.h"
-#include "vec/exec/ves_http_scanner.h"
namespace doris {
@@ -85,7 +84,6 @@ protected:
std::condition_variable _queue_reader_cond;
std::condition_variable _queue_writer_cond;
- bool _vectorized = false;
private:
// Create scanners to do scan job
@@ -98,10 +96,6 @@ private:
Status scanner_scan(std::unique_ptr<EsHttpScanner> scanner,
const std::vector<ExprContext*>& conjunct_ctxs, EsScanCounter* counter);
- virtual Status scanner_scan(std::unique_ptr<vectorized::VEsHttpScanner> scanner) {
- return Status::NotSupported("vectorized scan in EsHttpScanNode is not supported!");
- };
-
Status build_conjuncts_list();
std::vector<std::thread> _scanner_threads;
diff --git a/be/src/exec/exec_node.cpp b/be/src/exec/exec_node.cpp
index ce7bb4d317..9a67ced4cf 100644
--- a/be/src/exec/exec_node.cpp
+++ b/be/src/exec/exec_node.cpp
@@ -71,15 +71,11 @@
#include "vec/exec/vbroker_scan_node.h"
#include "vec/exec/vdata_gen_scan_node.h"
#include "vec/exec/vempty_set_node.h"
-#include "vec/exec/ves_http_scan_node.h"
#include "vec/exec/vexcept_node.h"
#include "vec/exec/vexchange_node.h"
#include "vec/exec/vintersect_node.h"
-#include "vec/exec/vjdbc_scan_node.h"
#include "vec/exec/vmysql_scan_node.h"
#include "vec/exec/vnested_loop_join_node.h"
-#include "vec/exec/vodbc_scan_node.h"
-#include "vec/exec/volap_scan_node.h"
#include "vec/exec/vrepeat_node.h"
#include "vec/exec/vschema_scan_node.h"
#include "vec/exec/vselect_node.h"
@@ -231,8 +227,7 @@ Status ExecNode::prepare(RuntimeState* state) {
// For vectorized olap scan node, the conjuncts is prepared in _vconjunct_ctx_ptr.
// And _conjunct_ctxs is useless.
// TODO: Should be removed when non-vec engine is removed.
- if (typeid(*this) != typeid(doris::vectorized::VOlapScanNode) &&
- typeid(*this) != typeid(doris::vectorized::NewOlapScanNode)) {
+ if (typeid(*this) != typeid(doris::vectorized::NewOlapScanNode)) {
RETURN_IF_ERROR(Expr::prepare(_conjunct_ctxs, state, _row_descriptor));
}
RETURN_IF_ERROR(vectorized::VExpr::prepare(_projections, state, _row_descriptor));
@@ -250,8 +245,7 @@ Status ExecNode::open(RuntimeState* state) {
RETURN_IF_ERROR((*_vconjunct_ctx_ptr)->open(state));
}
RETURN_IF_ERROR(vectorized::VExpr::open(_projections, state));
- if (typeid(*this) != typeid(doris::vectorized::VOlapScanNode) &&
- typeid(*this) != typeid(doris::vectorized::NewOlapScanNode)) {
+ if (typeid(*this) != typeid(doris::vectorized::NewOlapScanNode)) {
return Expr::open(_conjunct_ctxs, state);
} else {
return Status::OK();
@@ -295,8 +289,7 @@ Status ExecNode::close(RuntimeState* state) {
if (_vconjunct_ctx_ptr) {
(*_vconjunct_ctx_ptr)->close(state);
}
- if (typeid(*this) != typeid(doris::vectorized::VOlapScanNode) &&
- typeid(*this) != typeid(doris::vectorized::NewOlapScanNode)) {
+ if (typeid(*this) != typeid(doris::vectorized::NewOlapScanNode)) {
Expr::close(_conjunct_ctxs, state);
}
vectorized::VExpr::close(_projections, state);
@@ -449,11 +442,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
#endif
case TPlanNodeType::ODBC_SCAN_NODE:
if (state->enable_vectorized_exec()) {
- if (config::enable_new_scan_node) {
- *node = pool->add(new vectorized::NewOdbcScanNode(pool, tnode, descs));
- } else {
- *node = pool->add(new vectorized::VOdbcScanNode(pool, tnode, descs));
- }
+ *node = pool->add(new vectorized::NewOdbcScanNode(pool, tnode, descs));
} else {
*node = pool->add(new OdbcScanNode(pool, tnode, descs));
}
@@ -462,11 +451,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::JDBC_SCAN_NODE:
if (state->enable_vectorized_exec()) {
#ifdef LIBJVM
- if (config::enable_new_scan_node) {
- *node = pool->add(new vectorized::NewJdbcScanNode(pool, tnode, descs));
- } else {
- *node = pool->add(new vectorized::VJdbcScanNode(pool, tnode, descs));
- }
+ *node = pool->add(new vectorized::NewJdbcScanNode(pool, tnode, descs));
#else
return Status::InternalError("Jdbc scan node is disabled since no libjvm is found!");
#endif
@@ -477,11 +462,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::ES_HTTP_SCAN_NODE:
if (state->enable_vectorized_exec()) {
- if (config::enable_new_scan_node) {
- *node = pool->add(new vectorized::NewEsScanNode(pool, tnode, descs));
- } else {
- *node = pool->add(new vectorized::VEsHttpScanNode(pool, tnode, descs));
- }
+ *node = pool->add(new vectorized::NewEsScanNode(pool, tnode, descs));
} else {
*node = pool->add(new EsHttpScanNode(pool, tnode, descs));
}
@@ -497,11 +478,7 @@ Status ExecNode::create_node(RuntimeState* state, ObjectPool* pool, const TPlanN
case TPlanNodeType::OLAP_SCAN_NODE:
if (state->enable_vectorized_exec()) {
- if (config::enable_new_scan_node) {
- *node = pool->add(new vectorized::NewOlapScanNode(pool, tnode, descs));
- } else {
- *node = pool->add(new vectorized::VOlapScanNode(pool, tnode, descs));
- }
+ *node = pool->add(new vectorized::NewOlapScanNode(pool, tnode, descs));
} else {
*node = pool->add(new OlapScanNode(pool, tnode, descs));
}
diff --git a/be/src/exec/olap_scan_node.h b/be/src/exec/olap_scan_node.h
index 47cf6cd01c..ed018e2d93 100644
--- a/be/src/exec/olap_scan_node.h
+++ b/be/src/exec/olap_scan_node.h
@@ -30,7 +30,6 @@
#include "runtime/descriptors.h"
#include "util/progress_updater.h"
#include "util/spinlock.h"
-#include "vec/exec/volap_scanner.h"
namespace doris {
class IRuntimeFilter;
diff --git a/be/src/exprs/expr_context.h b/be/src/exprs/expr_context.h
index 06a7453540..e61d60fd00 100644
--- a/be/src/exprs/expr_context.h
+++ b/be/src/exprs/expr_context.h
@@ -35,10 +35,6 @@ USING_DORIS_UDF;
namespace doris {
-namespace vectorized {
-class VOlapScanNode;
-} // namespace vectorized
-
class Expr;
class MemPool;
class RuntimeState;
@@ -164,7 +160,6 @@ private:
friend class OlapScanNode;
friend class EsPredicate;
friend class RowGroupReader;
- friend class vectorized::VOlapScanNode;
/// FunctionContexts for each registered expression. The FunctionContexts are created
/// and owned by this ExprContext.
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 365b89aea3..61cd8e666d 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -88,11 +88,7 @@ set(VEC_FILES
data_types/data_type_jsonb.cpp
exec/vaggregation_node.cpp
exec/varrow_scanner.cpp
- exec/ves_http_scan_node.cpp
- exec/ves_http_scanner.cpp
- exec/volap_scan_node.cpp
exec/vsort_node.cpp
- exec/volap_scanner.cpp
exec/vexchange_node.cpp
exec/vset_operation_node.cpp
exec/vunion_node.cpp
@@ -100,7 +96,6 @@ set(VEC_FILES
exec/vexcept_node.cpp
exec/vselect_node.cpp
exec/vnested_loop_join_node.cpp
- exec/vodbc_scan_node.cpp
exec/vmysql_scan_node.cpp
exec/vschema_scan_node.cpp
exec/vempty_set_node.cpp
@@ -112,7 +107,6 @@ set(VEC_FILES
exec/vbroker_scanner.cpp
exec/vjson_scanner.cpp
exec/vjdbc_connector.cpp
- exec/vjdbc_scan_node.cpp
exec/vparquet_scanner.cpp
exec/vorc_scanner.cpp
exec/join/vhash_join_node.cpp
diff --git a/be/src/vec/exec/ves_http_scan_node.cpp b/be/src/vec/exec/ves_http_scan_node.cpp
deleted file mode 100644
index d0704a2477..0000000000
--- a/be/src/vec/exec/ves_http_scan_node.cpp
+++ /dev/null
@@ -1,447 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include "vec/exec/ves_http_scan_node.h"
-
-#include "exec/es/es_predicate.h"
-#include "exec/es/es_query_builder.h"
-#include "exec/es/es_scan_reader.h"
-#include "exec/es/es_scroll_query.h"
-#include "exprs/expr_context.h"
-#include "gen_cpp/PlanNodes_types.h"
-#include "runtime/runtime_state.h"
-#include "runtime/tuple_row.h"
-#include "util/runtime_profile.h"
-
-namespace doris::vectorized {
-
-VEsHttpScanNode::VEsHttpScanNode(ObjectPool* pool, const TPlanNode& tnode,
- const DescriptorTbl& descs)
- : ScanNode(pool, tnode, descs),
- _tuple_id(tnode.es_scan_node.tuple_id),
- _runtime_state(nullptr),
- _tuple_desc(nullptr),
- _num_running_scanners(0),
- _scan_finished(false),
- _eos(false),
- _max_buffered_batches(1024),
- _wait_scanner_timer(nullptr) {}
-
-Status VEsHttpScanNode::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(ScanNode::init(tnode, state));
-
- // use TEsScanNode
- _properties = tnode.es_scan_node.properties;
-
- if (tnode.es_scan_node.__isset.docvalue_context) {
- _docvalue_context = tnode.es_scan_node.docvalue_context;
- }
-
- if (tnode.es_scan_node.__isset.fields_context) {
- _fields_context = tnode.es_scan_node.fields_context;
- }
- return Status::OK();
-}
-
-Status VEsHttpScanNode::prepare(RuntimeState* state) {
- VLOG_QUERY << "VEsHttpScanNode prepare";
- RETURN_IF_ERROR(ScanNode::prepare(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
-
- _scanner_profile.reset(new RuntimeProfile("EsHttpScanNode"));
- runtime_profile()->add_child(_scanner_profile.get(), true, nullptr);
-
- _runtime_state = state;
- _tuple_desc = state->desc_tbl().get_tuple_descriptor(_tuple_id);
- if (_tuple_desc == nullptr) {
- return Status::InternalError("Failed to get tuple descriptor, _tuple_id=i{}", _tuple_id);
- }
-
- // set up column name vector for ESScrollQueryBuilder
- for (auto slot_desc : _tuple_desc->slots()) {
- if (!slot_desc->is_materialized()) {
- continue;
- }
- _column_names.push_back(slot_desc->col_name());
- }
-
- _wait_scanner_timer = ADD_TIMER(runtime_profile(), "WaitScannerTime");
-
- return Status::OK();
-}
-
-// build predicate
-Status VEsHttpScanNode::build_conjuncts_list() {
- Status status = Status::OK();
- _conjunct_to_predicate.resize(_conjunct_ctxs.size());
-
- for (int i = 0; i < _conjunct_ctxs.size(); ++i) {
- EsPredicate* predicate = _pool->add(new EsPredicate(_conjunct_ctxs[i], _tuple_desc, _pool));
- predicate->set_field_context(_fields_context);
- status = predicate->build_disjuncts_list();
- if (status.ok()) {
- _conjunct_to_predicate[i] = _predicate_to_conjunct.size();
- _predicate_to_conjunct.push_back(i);
-
- _predicates.push_back(predicate);
- } else {
- _conjunct_to_predicate[i] = -1;
-
- VLOG_CRITICAL << status.get_error_msg();
- status = predicate->get_es_query_status();
- if (!status.ok()) {
- LOG(WARNING) << status.get_error_msg();
- return status;
- }
- }
- }
-
- return Status::OK();
-}
-
-Status VEsHttpScanNode::open(RuntimeState* state) {
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VEsHttpScanNode::open");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- RETURN_IF_ERROR(ExecNode::open(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
- RETURN_IF_CANCELLED(state);
-
- // fe by enable_new_es_dsl to control whether to generate DSL for easy rollback. After the code is stable, can delete the be generation logic
- if (_properties.find(ESScanReader::KEY_QUERY_DSL) != _properties.end()) {
- RETURN_IF_ERROR(start_scanners());
- return Status::OK();
- }
-
- // if conjunct is constant, compute direct and set eos = true
- for (int conj_idx = 0; conj_idx < _conjunct_ctxs.size(); ++conj_idx) {
- if (_conjunct_ctxs[conj_idx]->root()->is_constant()) {
- void* value = _conjunct_ctxs[conj_idx]->get_value(nullptr);
- if (value == nullptr || *reinterpret_cast<bool*>(value) == false) {
- _eos = true;
- }
- }
- }
-
- RETURN_IF_ERROR(build_conjuncts_list());
-
- // remove those predicates which ES cannot support
- std::vector<bool> list;
- BooleanQueryBuilder::validate(_predicates, &list);
-
- DCHECK(list.size() == _predicate_to_conjunct.size());
- for (int i = list.size() - 1; i >= 0; i--) {
- if (!list[i]) {
- _predicate_to_conjunct.erase(_predicate_to_conjunct.begin() + i);
- _predicates.erase(_predicates.begin() + i);
- }
- }
-
- // filter the conjuncts and ES will process them later
- for (int i = _predicate_to_conjunct.size() - 1; i >= 0; i--) {
- int conjunct_index = _predicate_to_conjunct[i];
- _conjunct_ctxs[conjunct_index]->close(_runtime_state);
- _conjunct_ctxs.erase(_conjunct_ctxs.begin() + conjunct_index);
- }
-
- auto checker = [&](int index) {
- return _conjunct_to_predicate[index] != -1 && list[_conjunct_to_predicate[index]];
- };
- _peel_pushed_vconjunct(state, checker);
-
- RETURN_IF_ERROR(start_scanners());
-
- return Status::OK();
-}
-
-// Prefer to the local host
-static std::string get_host_port(const std::vector<TNetworkAddress>& es_hosts) {
- std::string host_port;
- std::string localhost = BackendOptions::get_localhost();
-
- TNetworkAddress host = es_hosts[0];
- for (auto& es_host : es_hosts) {
- if (es_host.hostname == localhost) {
- host = es_host;
- break;
- }
- }
-
- host_port = host.hostname;
- host_port += ":";
- host_port += std::to_string(host.port);
- return host_port;
-}
-
-Status VEsHttpScanNode::start_scanners() {
- {
- std::unique_lock<std::mutex> l(_batch_queue_lock);
- _num_running_scanners = _scan_ranges.size();
- }
-
- _scanners_status.resize(_scan_ranges.size());
- for (int i = 0; i < _scan_ranges.size(); i++) {
- _scanner_threads.emplace_back(
- [this, i, length = _scan_ranges.size(), &p_status = _scanners_status[i],
- parent_span = opentelemetry::trace::Tracer::GetCurrentSpan()] {
- OpentelemetryScope scope {parent_span};
- this->scanner_worker(i, length, p_status);
- });
- }
- return Status::OK();
-}
-
-Status VEsHttpScanNode::get_next(RuntimeState* state, vectorized::Block* block, bool* eos) {
- INIT_AND_SCOPE_GET_NEXT_SPAN(state->get_tracer(), _get_next_span, "VEsHttpScanNode::get_next");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- if (state->is_cancelled()) {
- std::unique_lock<std::mutex> l(_block_queue_lock);
- if (update_status(Status::Cancelled("Cancelled"))) {
- _queue_writer_cond.notify_all();
- }
- }
-
- if (_scan_finished.load() || _eos) {
- *eos = true;
- return Status::OK();
- }
-
- std::shared_ptr<vectorized::Block> scanner_block;
- {
- std::unique_lock<std::mutex> l(_block_queue_lock);
- while (_process_status.ok() && !_runtime_state->is_cancelled() &&
- _num_running_scanners > 0 && _block_queue.empty()) {
- SCOPED_TIMER(_wait_scanner_timer);
- _queue_reader_cond.wait_for(l, std::chrono::seconds(1));
- }
- if (!_process_status.ok()) {
- // Some scanner process failed.
- return _process_status;
- }
- if (_runtime_state->is_cancelled()) {
- if (update_status(Status::Cancelled("Cancelled"))) {
- _queue_writer_cond.notify_all();
- }
- return _process_status;
- }
- if (!_block_queue.empty()) {
- scanner_block = _block_queue.front();
- _block_queue.pop_front();
- }
- }
-
- // All scanner has been finished, and all cached batch has been read
- if (scanner_block == nullptr) {
- _scan_finished.store(true);
- *eos = true;
- return Status::OK();
- }
-
- // notify one scanner
- _queue_writer_cond.notify_one();
-
- reached_limit(scanner_block.get(), eos);
- *block = *scanner_block;
-
- // This is first time reach limit.
- // Only valid when query 'select * from table1 limit 20'
- if (*eos) {
- _scan_finished.store(true);
- _queue_writer_cond.notify_all();
- LOG(INFO) << "VEsHttpScanNode ReachedLimit.";
- *eos = true;
- } else {
- *eos = false;
- }
-
- return Status::OK();
-}
-
-Status VEsHttpScanNode::scanner_scan(std::unique_ptr<VEsHttpScanner> scanner) {
- RETURN_IF_ERROR(scanner->open());
- bool scanner_eof = false;
-
- const int batch_size = _runtime_state->batch_size();
- std::unique_ptr<MemPool> tuple_pool(new MemPool(mem_tracker()));
- size_t slot_num = _tuple_desc->slots().size();
-
- while (!scanner_eof) {
- std::shared_ptr<vectorized::Block> block(new vectorized::Block());
- std::vector<vectorized::MutableColumnPtr> columns(slot_num);
- for (int i = 0; i < slot_num; i++) {
- columns[i] = _tuple_desc->slots()[i]->get_empty_mutable_column();
- }
- while (columns[0]->size() < batch_size && !scanner_eof) {
- RETURN_IF_CANCELLED(_runtime_state);
-
- // If we have finished all works
- if (_scan_finished.load()) {
- return Status::OK();
- }
-
- // Get from scanner
- RETURN_IF_ERROR(
- scanner->get_next(columns, tuple_pool.get(), &scanner_eof, _docvalue_context));
- }
-
- if (columns[0]->size() > 0) {
- auto n_columns = 0;
- for (const auto slot_desc : _tuple_desc->slots()) {
- block->insert(ColumnWithTypeAndName(std::move(columns[n_columns++]),
- slot_desc->get_data_type_ptr(),
- slot_desc->col_name()));
- }
-
- RETURN_IF_ERROR(VExprContext::filter_block(_vconjunct_ctx_ptr, block.get(),
- _tuple_desc->slots().size()));
-
- std::unique_lock<std::mutex> l(_block_queue_lock);
- while (_process_status.ok() && !_scan_finished.load() &&
- !_runtime_state->is_cancelled() &&
- _block_queue.size() >= _max_buffered_batches) {
- _queue_writer_cond.wait_for(l, std::chrono::seconds(1));
- }
- // Process already set failed, so we just return OK
- if (!_process_status.ok()) {
- return Status::OK();
- }
- // Scan already finished, just return
- if (_scan_finished.load()) {
- return Status::OK();
- }
- // Runtime state is canceled, just return cancel
- if (_runtime_state->is_cancelled()) {
- return Status::Cancelled("Cancelled");
- }
- _block_queue.push_back(block);
-
- // Notify reader to process
- _queue_reader_cond.notify_one();
- }
- }
-
- return Status::OK();
-}
-
-Status VEsHttpScanNode::close(RuntimeState* state) {
- if (is_closed()) {
- return Status::OK();
- }
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- _scan_finished.store(true);
- _queue_writer_cond.notify_all();
- _queue_reader_cond.notify_all();
- for (int i = 0; i < _scanner_threads.size(); ++i) {
- _scanner_threads[i].join();
- }
-
- //don't need to hold lock to update_status in close function
- //collect scanners status
- update_status(collect_scanners_status());
-
- _batch_queue.clear();
- _block_queue.clear();
-
- RETURN_IF_ERROR(ScanNode::close(state));
- return _process_status;
-}
-
-Status VEsHttpScanNode::collect_scanners_status() {
- // NOTE. if open() was called, but set_range() was NOT called for some reason.
- // then close() was called.
- // there would cause a core because _scanners_status's iterator was in [0, _scan_ranges) other than [0, _scanners_status)
- // it is said that the fragment-call-frame is calling scan-node in this way....
- // in my options, it's better fixed in fragment-call-frame. e.g. call close() according the return value of open()
- for (int i = 0; i < _scanners_status.size(); i++) {
- std::future<Status> f = _scanners_status[i].get_future();
- RETURN_IF_ERROR(f.get());
- }
- return Status::OK();
-}
-
-// This function is called after plan node has been prepared.
-Status VEsHttpScanNode::set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) {
- _scan_ranges = scan_ranges;
- return Status::OK();
-}
-
-void VEsHttpScanNode::debug_string(int ident_level, std::stringstream* out) const {
- (*out) << "VEsHttpScanNode";
-}
-
-void VEsHttpScanNode::scanner_worker(int start_idx, int length, std::promise<Status>& p_status) {
- START_AND_SCOPE_SPAN(_runtime_state->get_tracer(), span, "VEsHttpScanNode::scanner_worker");
- SCOPED_ATTACH_TASK(_runtime_state);
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker_shared());
- // Clone expr context
- std::vector<ExprContext*> scanner_expr_ctxs;
- DCHECK(start_idx < length);
- auto status = Expr::clone_if_not_exists(_conjunct_ctxs, _runtime_state, &scanner_expr_ctxs);
- if (!status.ok()) {
- LOG(WARNING) << "Clone conjuncts failed.";
- }
-
- EsScanCounter counter;
- const TEsScanRange& es_scan_range = _scan_ranges[start_idx].scan_range.es_scan_range;
-
- // Collect the information from scan range to properties
- std::map<std::string, std::string> properties(_properties);
- properties[ESScanReader::KEY_INDEX] = es_scan_range.index;
- if (es_scan_range.__isset.type) {
- properties[ESScanReader::KEY_TYPE] = es_scan_range.type;
- }
- properties[ESScanReader::KEY_SHARD] = std::to_string(es_scan_range.shard_id);
- properties[ESScanReader::KEY_BATCH_SIZE] = std::to_string(_runtime_state->batch_size());
- properties[ESScanReader::KEY_HOST_PORT] = get_host_port(es_scan_range.es_hosts);
- // push down limit to Elasticsearch
- // if predicate in _conjunct_ctxs can not be processed by Elasticsearch, we can not push down limit operator to Elasticsearch
- if (limit() != -1 && limit() <= _runtime_state->batch_size() && _conjunct_ctxs.empty()) {
- properties[ESScanReader::KEY_TERMINATE_AFTER] = std::to_string(limit());
- }
-
- bool doc_value_mode = false;
- properties[ESScanReader::KEY_QUERY] = ESScrollQueryBuilder::build(
- properties, _column_names, _predicates, _docvalue_context, &doc_value_mode);
-
- // start scanner to scan
- std::unique_ptr<vectorized::VEsHttpScanner> scanner(
- new vectorized::VEsHttpScanner(_runtime_state, runtime_profile(), _tuple_id, properties,
- scanner_expr_ctxs, &counter, doc_value_mode));
- status = scanner_scan(std::move(scanner));
- if (!status.ok()) {
- LOG(WARNING) << "Scanner[" << start_idx
- << "] process failed. status=" << status.get_error_msg();
- }
-
- // scanner is going to finish
- {
- std::lock_guard<std::mutex> l(_batch_queue_lock);
- if (!status.ok()) {
- update_status(status);
- }
- // This scanner will finish
- _num_running_scanners--;
- }
- _queue_reader_cond.notify_all();
- // If one scanner failed, others don't need scan any more
- if (!status.ok()) {
- _queue_writer_cond.notify_all();
- }
-
- p_status.set_value(status);
-}
-
-} // namespace doris::vectorized
diff --git a/be/src/vec/exec/ves_http_scan_node.h b/be/src/vec/exec/ves_http_scan_node.h
deleted file mode 100644
index cd42a573e4..0000000000
--- a/be/src/vec/exec/ves_http_scan_node.h
+++ /dev/null
@@ -1,113 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include <memory>
-
-#include "exec/es_http_scan_node.h"
-#include "exec/scan_node.h"
-#include "runtime/descriptors.h"
-#include "vec/exec/ves_http_scanner.h"
-
-namespace doris {
-
-class RuntimeState;
-class Status;
-
-namespace vectorized {
-
-class VEsHttpScanNode : public ScanNode {
-public:
- VEsHttpScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs);
- ~VEsHttpScanNode() = default;
-
- virtual Status get_next(RuntimeState* state, vectorized::Block* block, bool* eos) override;
-
- virtual Status init(const TPlanNode& tnode, RuntimeState* state = nullptr) override;
- virtual Status prepare(RuntimeState* state) override;
- virtual Status open(RuntimeState* state) override;
- Status get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) override {
- return Status::NotSupported("Not Implemented VEsHttpScanNode Node::get_next scalar");
- }
- virtual Status close(RuntimeState* state) override;
- virtual Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) override;
-
-private:
- // Write debug string of this into out.
- virtual void debug_string(int indentation_level, std::stringstream* out) const override;
-
- // Update process status to one failed status,
- // NOTE: Must hold the mutex of this scan node
- bool update_status(const Status& new_status) {
- if (_process_status.ok()) {
- _process_status = new_status;
- return true;
- }
- return false;
- }
- // One scanner worker, This scanner will handle 'length' ranges start from start_idx
- virtual void scanner_worker(int start_idx, int length, std::promise<Status>& p_status);
-
- TupleId _tuple_id;
- RuntimeState* _runtime_state;
- TupleDescriptor* _tuple_desc;
-
- int _num_running_scanners;
- std::atomic<bool> _scan_finished;
- bool _eos;
- int _max_buffered_batches;
- RuntimeProfile::Counter* _wait_scanner_timer;
-
- Status _process_status;
-
- std::map<std::string, std::string> _docvalue_context;
-
- std::condition_variable _queue_reader_cond;
- std::condition_variable _queue_writer_cond;
-
- // Create scanners to do scan job
- Status start_scanners();
-
- // Collect all scanners 's status
- Status collect_scanners_status();
-
- Status build_conjuncts_list();
-
- std::vector<std::thread> _scanner_threads;
- std::vector<std::promise<Status>> _scanners_status;
- std::map<std::string, std::string> _properties;
- std::map<std::string, std::string> _fields_context;
- std::vector<TScanRangeParams> _scan_ranges;
- std::vector<std::string> _column_names;
-
- std::mutex _batch_queue_lock;
- std::deque<std::shared_ptr<RowBatch>> _batch_queue;
- std::vector<EsPredicate*> _predicates;
-
- std::vector<int> _predicate_to_conjunct;
- std::vector<int> _conjunct_to_predicate;
-
- std::unique_ptr<RuntimeProfile> _scanner_profile;
-
- Status scanner_scan(std::unique_ptr<VEsHttpScanner> scanner);
-
- std::deque<std::shared_ptr<vectorized::Block>> _block_queue;
- std::mutex _block_queue_lock;
-};
-} // namespace vectorized
-} // namespace doris
\ No newline at end of file
diff --git a/be/src/vec/exec/ves_http_scanner.cpp b/be/src/vec/exec/ves_http_scanner.cpp
deleted file mode 100644
index 11e3e87f94..0000000000
--- a/be/src/vec/exec/ves_http_scanner.cpp
+++ /dev/null
@@ -1,52 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include "vec/exec/ves_http_scanner.h"
-
-namespace doris::vectorized {
-
-Status VEsHttpScanner::get_next(std::vector<vectorized::MutableColumnPtr>& columns,
- MemPool* tuple_pool, bool* eof,
- const std::map<std::string, std::string>& docvalue_context) {
- SCOPED_TIMER(_read_timer);
- if (_line_eof && _batch_eof) {
- *eof = true;
- return Status::OK();
- }
-
- while (!_batch_eof) {
- if (_line_eof || _es_scroll_parser == nullptr) {
- RETURN_IF_ERROR(_es_reader->get_next(&_batch_eof, _es_scroll_parser));
- if (_batch_eof) {
- *eof = true;
- return Status::OK();
- }
- }
-
- COUNTER_UPDATE(_rows_read_counter, 1);
- SCOPED_TIMER(_materialize_timer);
- RETURN_IF_ERROR(_es_scroll_parser->fill_columns(_tuple_desc, columns, tuple_pool,
- &_line_eof, docvalue_context));
- if (!_line_eof) {
- break;
- }
- }
-
- return Status::OK();
-}
-
-} // namespace doris::vectorized
diff --git a/be/src/vec/exec/ves_http_scanner.h b/be/src/vec/exec/ves_http_scanner.h
deleted file mode 100644
index a315e3e776..0000000000
--- a/be/src/vec/exec/ves_http_scanner.h
+++ /dev/null
@@ -1,38 +0,0 @@
-
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include <exec/es_http_scanner.h>
-
-namespace doris::vectorized {
-
-class VEsHttpScanner : public EsHttpScanner {
-public:
- VEsHttpScanner(RuntimeState* state, RuntimeProfile* profile, TupleId tuple_id,
- const std::map<std::string, std::string>& properties,
- const std::vector<ExprContext*>& conjunct_ctxs, EsScanCounter* counter,
- bool doc_value_mode)
- : EsHttpScanner(state, profile, tuple_id, properties, conjunct_ctxs, counter,
- doc_value_mode) {};
-
- Status get_next(std::vector<vectorized::MutableColumnPtr>& columns, MemPool* tuple_pool,
- bool* eof, const std::map<std::string, std::string>& docvalue_context);
-};
-
-} // namespace doris::vectorized
diff --git a/be/src/vec/exec/vjdbc_connector.cpp b/be/src/vec/exec/vjdbc_connector.cpp
index 5cff86bf46..283aa3e2a7 100644
--- a/be/src/vec/exec/vjdbc_connector.cpp
+++ b/be/src/vec/exec/vjdbc_connector.cpp
@@ -410,4 +410,4 @@ FUNC_IMPL_TO_CONVERT_DATA(double, double, D, Double)
} // namespace vectorized
} // namespace doris
-#endif
\ No newline at end of file
+#endif
diff --git a/be/src/vec/exec/vjdbc_connector.h b/be/src/vec/exec/vjdbc_connector.h
index e6f957746f..ba2f19c7b8 100644
--- a/be/src/vec/exec/vjdbc_connector.h
+++ b/be/src/vec/exec/vjdbc_connector.h
@@ -105,4 +105,4 @@ private:
} // namespace vectorized
} // namespace doris
-#endif
\ No newline at end of file
+#endif
diff --git a/be/src/vec/exec/vjdbc_scan_node.cpp b/be/src/vec/exec/vjdbc_scan_node.cpp
deleted file mode 100644
index 7d15603992..0000000000
--- a/be/src/vec/exec/vjdbc_scan_node.cpp
+++ /dev/null
@@ -1,173 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-#include "vec/exec/vjdbc_scan_node.h"
-#ifdef LIBJVM
-#include <string>
-
-#include "common/status.h"
-
-namespace doris {
-namespace vectorized {
-
-VJdbcScanNode::VJdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs)
- : ScanNode(pool, tnode, descs),
- _is_init(false),
- _table_name(tnode.jdbc_scan_node.table_name),
- _tuple_id(tnode.jdbc_scan_node.tuple_id),
- _query_string(tnode.jdbc_scan_node.query_string),
- _tuple_desc(nullptr) {}
-
-Status VJdbcScanNode::prepare(RuntimeState* state) {
- VLOG_CRITICAL << "VJdbcScanNode::Prepare";
- if (_is_init) {
- return Status::OK();
- }
-
- if (state == nullptr) {
- return Status::InternalError("input pointer is NULL of VJdbcScanNode::prepare.");
- }
-
- RETURN_IF_ERROR(ScanNode::prepare(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
-
- // get tuple desc
- _tuple_desc = state->desc_tbl().get_tuple_descriptor(_tuple_id);
- if (_tuple_desc == nullptr) {
- return Status::InternalError("Failed to get tuple descriptor.");
- }
-
- // get jdbc table info
- const JdbcTableDescriptor* jdbc_table =
- static_cast<const JdbcTableDescriptor*>(_tuple_desc->table_desc());
- if (jdbc_table == nullptr) {
- return Status::InternalError("jdbc table pointer is NULL of VJdbcScanNode::prepare.");
- }
- _jdbc_param.driver_class = jdbc_table->jdbc_driver_class();
- _jdbc_param.driver_path = jdbc_table->jdbc_driver_url();
- _jdbc_param.resource_name = jdbc_table->jdbc_resource_name();
- _jdbc_param.driver_checksum = jdbc_table->jdbc_driver_checksum();
- _jdbc_param.jdbc_url = jdbc_table->jdbc_url();
- _jdbc_param.user = jdbc_table->jdbc_user();
- _jdbc_param.passwd = jdbc_table->jdbc_passwd();
- _jdbc_param.tuple_desc = _tuple_desc;
- _jdbc_param.query_string = std::move(_query_string);
-
- _jdbc_connector.reset(new (std::nothrow) JdbcConnector(_jdbc_param));
- if (_jdbc_connector == nullptr) {
- return Status::InternalError("new a jdbc scanner failed.");
- }
-
- _is_init = true;
- return Status::OK();
-}
-
-Status VJdbcScanNode::open(RuntimeState* state) {
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VJdbcScanNode::open");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- RETURN_IF_ERROR(ExecNode::open(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
- VLOG_CRITICAL << "VJdbcScanNode::open";
-
- if (state == nullptr) {
- return Status::InternalError("input pointer is NULL of VJdbcScanNode::open.");
- }
-
- if (!_is_init) {
- return Status::InternalError("used before initialize of VJdbcScanNode::open.");
- }
-
- RETURN_IF_CANCELLED(state);
- RETURN_IF_ERROR(_jdbc_connector->open(state, true));
- RETURN_IF_ERROR(_jdbc_connector->query());
- return Status::OK();
-}
-
-Status VJdbcScanNode::get_next(RuntimeState* state, vectorized::Block* block, bool* eos) {
- VLOG_CRITICAL << "VJdbcScanNode::get_next";
- INIT_AND_SCOPE_GET_NEXT_SPAN(state->get_tracer(), _get_next_span, "VJdbcScanNode::get_next");
- if (nullptr == state || nullptr == block || nullptr == eos) {
- return Status::InternalError("input is NULL pointer");
- }
-
- if (!_is_init) {
- return Status::InternalError("used before initialize of VJdbcScanNode::get_next.");
- }
-
- auto column_size = _tuple_desc->slots().size();
- std::vector<MutableColumnPtr> columns(column_size);
- bool mem_reuse = block->mem_reuse();
- // only empty block should be here
- DCHECK(block->rows() == 0);
-
- bool jdbc_eos = false;
- do {
- RETURN_IF_CANCELLED(state);
-
- columns.resize(column_size);
- for (auto i = 0; i < column_size; i++) {
- if (mem_reuse) {
- columns[i] = std::move(*block->get_by_position(i).column).mutate();
- } else {
- columns[i] = _tuple_desc->slots()[i]->get_empty_mutable_column();
- }
- }
-
- RETURN_IF_ERROR(_jdbc_connector->get_next(&jdbc_eos, columns, state->batch_size()));
-
- if (jdbc_eos) {
- *eos = true;
- break;
- }
-
- // Before really use the Block, must clear other ptr of column in block
- // So here need do std::move and clear in `columns`
- if (!mem_reuse) {
- int column_index = 0;
- for (const auto slot_desc : _tuple_desc->slots()) {
- block->insert(ColumnWithTypeAndName(std::move(columns[column_index++]),
- slot_desc->get_data_type_ptr(),
- slot_desc->col_name()));
- }
- } else {
- columns.clear();
- }
- VLOG_ROW << "VJdbcScanNode output rows: " << block->rows();
- } while (block->rows() == 0 && !(*eos));
-
- RETURN_IF_ERROR(VExprContext::filter_block(_vconjunct_ctx_ptr, block, block->columns()));
- reached_limit(block, eos);
- return Status::OK();
-}
-
-Status VJdbcScanNode::close(RuntimeState* state) {
- if (is_closed()) {
- return Status::OK();
- }
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VJdbcScanNode::close");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- return ExecNode::close(state);
-}
-
-// No use
-Status VJdbcScanNode::set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) {
- return Status::OK();
-}
-
-} // namespace vectorized
-} // namespace doris
-
-#endif
\ No newline at end of file
diff --git a/be/src/vec/exec/vjdbc_scan_node.h b/be/src/vec/exec/vjdbc_scan_node.h
deleted file mode 100644
index c9f13a544e..0000000000
--- a/be/src/vec/exec/vjdbc_scan_node.h
+++ /dev/null
@@ -1,68 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-#ifdef LIBJVM
-
-#include "exec/exec_node.h"
-#include "exec/scan_node.h"
-#include "vec/exec/vjdbc_connector.h"
-namespace doris {
-
-namespace vectorized {
-class VJdbcScanNode final : public ScanNode {
-public:
- VJdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs);
-
- ~VJdbcScanNode() override = default;
-
- Status prepare(RuntimeState* state) override;
-
- Status open(RuntimeState* state) override;
-
- Status get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) override {
- return Status::NotSupported("Not Implemented JdbcScanNode::get_next.");
- }
-
- Status get_next(RuntimeState* state, vectorized::Block* block, bool* eos) override;
-
- Status close(RuntimeState* state) override;
-
- // No use
- Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) override;
-
-private:
- std::string get_query_stmt(const std::string& table, const std::vector<std::string>& fields,
- const std::vector<std::string>& filters, int64_t limit);
-
- bool _is_init;
- std::string _table_name;
-
- // Tuple id resolved in prepare() to set _tuple_desc;
- TupleId _tuple_id;
- //SQL
- std::string _query_string;
- // Descriptor of tuples read from JDBC table.
- const TupleDescriptor* _tuple_desc;
-
- // Scanner of JDBC.
- std::unique_ptr<JdbcConnector> _jdbc_connector;
- JdbcConnectorParam _jdbc_param;
-};
-} // namespace vectorized
-} // namespace doris
-#endif
\ No newline at end of file
diff --git a/be/src/vec/exec/vodbc_scan_node.cpp b/be/src/vec/exec/vodbc_scan_node.cpp
deleted file mode 100644
index 994eedc926..0000000000
--- a/be/src/vec/exec/vodbc_scan_node.cpp
+++ /dev/null
@@ -1,249 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include "vec/exec/vodbc_scan_node.h"
-
-#include "exec/text_converter.h"
-#include "exec/text_converter.hpp"
-
-namespace doris {
-namespace vectorized {
-
-VOdbcScanNode::VOdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs,
- std::string scan_node_type)
- : ScanNode(pool, tnode, descs),
- _is_init(false),
- _scan_node_type(std::move(scan_node_type)),
- _table_name(tnode.odbc_scan_node.table_name),
- _connect_string(std::move(tnode.odbc_scan_node.connect_string)),
- _query_string(std::move(tnode.odbc_scan_node.query_string)),
- _tuple_id(tnode.odbc_scan_node.tuple_id),
- _tuple_desc(nullptr),
- _slot_num(0) {}
-
-Status VOdbcScanNode::prepare(RuntimeState* state) {
- VLOG_CRITICAL << _scan_node_type << "::Prepare";
-
- if (_is_init) {
- return Status::OK();
- }
-
- if (nullptr == state) {
- return Status::InternalError("input pointer is null.");
- }
-
- RETURN_IF_ERROR(ScanNode::prepare(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
- // get tuple desc
- _tuple_desc = state->desc_tbl().get_tuple_descriptor(_tuple_id);
-
- if (nullptr == _tuple_desc) {
- return Status::InternalError("Failed to get tuple descriptor.");
- }
-
- _slot_num = _tuple_desc->slots().size();
-
- _odbc_param.connect_string = std::move(_connect_string);
- _odbc_param.query_string = std::move(_query_string);
- _odbc_param.tuple_desc = _tuple_desc;
-
- _odbc_scanner.reset(new (std::nothrow) ODBCConnector(_odbc_param));
-
- if (_odbc_scanner.get() == nullptr) {
- return Status::InternalError("new a odbc scanner failed.");
- }
-
- _tuple_pool.reset(new (std::nothrow) MemPool());
-
- if (_tuple_pool.get() == nullptr) {
- return Status::InternalError("new a mem pool failed.");
- }
-
- _text_converter.reset(new (std::nothrow) TextConverter('\\'));
-
- if (_text_converter.get() == nullptr) {
- return Status::InternalError("new a text convertor failed.");
- }
-
- _is_init = true;
-
- return Status::OK();
-}
-
-Status VOdbcScanNode::open(RuntimeState* state) {
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VOdbcScanNode::open");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- RETURN_IF_ERROR(ExecNode::open(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
- VLOG_CRITICAL << _scan_node_type << "::Open";
-
- if (nullptr == state) {
- return Status::InternalError("input pointer is null.");
- }
-
- if (!_is_init) {
- return Status::InternalError("used before initialize.");
- }
-
- RETURN_IF_CANCELLED(state);
- RETURN_IF_ERROR(_odbc_scanner->open(state));
- RETURN_IF_ERROR(_odbc_scanner->query());
- // check materialize slot num
-
- return Status::OK();
-}
-
-Status VOdbcScanNode::get_next(RuntimeState* state, Block* block, bool* eos) {
- INIT_AND_SCOPE_GET_NEXT_SPAN(state->get_tracer(), _get_next_span, "VOdbcScanNode::get_next");
- VLOG_CRITICAL << get_scan_node_type() << "::GetNext";
-
- if (nullptr == state || nullptr == block || nullptr == eos) {
- return Status::InternalError("input is NULL pointer");
- }
-
- if (!is_init()) {
- return Status::InternalError("used before initialize.");
- }
- RETURN_IF_CANCELLED(state);
-
- auto odbc_scanner = get_odbc_scanner();
- auto tuple_desc = get_tuple_desc();
- auto text_converter = get_text_converter();
-
- auto column_size = tuple_desc->slots().size();
- std::vector<MutableColumnPtr> columns(column_size);
-
- bool mem_reuse = block->mem_reuse();
- // only empty block should be here
- DCHECK(block->rows() == 0);
-
- // Indicates whether there are more rows to process. Set in _odbc_scanner.next().
- bool odbc_eos = false;
-
- do {
- RETURN_IF_CANCELLED(state);
-
- columns.resize(column_size);
- for (auto i = 0; i < column_size; i++) {
- if (mem_reuse) {
- columns[i] = std::move(*block->get_by_position(i).column).mutate();
- } else {
- columns[i] = tuple_desc->slots()[i]->get_empty_mutable_column();
- }
- }
-
- while (true) {
- // block is full, break
- if (state->batch_size() <= columns[0]->size()) {
- break;
- }
-
- RETURN_IF_ERROR(odbc_scanner->get_next_row(&odbc_eos));
-
- if (odbc_eos) {
- *eos = true;
- break;
- }
-
- // Read one row from reader
- for (int column_index = 0, materialized_column_index = 0; column_index < column_size;
- ++column_index) {
- auto slot_desc = tuple_desc->slots()[column_index];
- // because the fe planner filter the non_materialize column
- if (!slot_desc->is_materialized()) {
- continue;
- }
- const auto& column_data = odbc_scanner->get_column_data(materialized_column_index);
-
- char* value_data = static_cast<char*>(column_data.target_value_ptr);
- int value_len = column_data.strlen_or_ind;
-
- if (value_len == SQL_NULL_DATA) {
- if (slot_desc->is_nullable()) {
- columns[column_index]->insert_default();
- } else {
- return Status::InternalError(
- "nonnull column contains nullptr. table={}, column={}", _table_name,
- slot_desc->col_name());
- }
- } else if (value_len > column_data.buffer_length) {
- return Status::InternalError(
- "column value length longer than buffer length. "
- "table={}, column={}, buffer_length",
- _table_name, slot_desc->col_name(), column_data.buffer_length);
- } else {
- if (!text_converter->write_column(slot_desc, &columns[column_index], value_data,
- value_len, true, false)) {
- std::stringstream ss;
- ss << "Fail to convert odbc value:'" << value_data << "' to "
- << slot_desc->type() << " on column:`" << slot_desc->col_name() + "`";
- return Status::InternalError(ss.str());
- }
- }
- materialized_column_index++;
- }
- }
-
- // Before really use the Block, muse clear other ptr of column in block
- // So here need do std::move and clear in `columns`
- if (!mem_reuse) {
- int column_index = 0;
- for (const auto slot_desc : tuple_desc->slots()) {
- block->insert(ColumnWithTypeAndName(std::move(columns[column_index++]),
- slot_desc->get_data_type_ptr(),
- slot_desc->col_name()));
- }
- } else {
- columns.clear();
- }
- VLOG_ROW << "VOdbcScanNode output rows: " << block->rows();
- } while (block->rows() == 0 && !(*eos));
-
- RETURN_IF_ERROR(VExprContext::filter_block(_vconjunct_ctx_ptr, block, block->columns()));
- reached_limit(block, eos);
-
- return Status::OK();
-}
-
-Status VOdbcScanNode::close(RuntimeState* state) {
- if (is_closed()) {
- return Status::OK();
- }
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VOdbcScanNode::close");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
-
- _tuple_pool.reset();
-
- return ExecNode::close(state);
-}
-
-void VOdbcScanNode::debug_string(int indentation_level, std::stringstream* out) const {
- *out << string(indentation_level * 2, ' ');
- *out << _scan_node_type << "(tupleid=" << _tuple_id << " table=" << _table_name;
- *out << ")" << std::endl;
-
- for (int i = 0; i < _children.size(); ++i) {
- _children[i]->debug_string(indentation_level + 1, out);
- }
-}
-
-Status VOdbcScanNode::set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) {
- return Status::OK();
-}
-
-} // namespace vectorized
-} // namespace doris
diff --git a/be/src/vec/exec/vodbc_scan_node.h b/be/src/vec/exec/vodbc_scan_node.h
deleted file mode 100644
index 6261bbed9c..0000000000
--- a/be/src/vec/exec/vodbc_scan_node.h
+++ /dev/null
@@ -1,87 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include "exec/odbc_connector.h"
-#include "exec/scan_node.h"
-#include "exec/text_converter.hpp"
-
-namespace doris {
-namespace vectorized {
-
-class VOdbcScanNode : public ScanNode {
-public:
- VOdbcScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs,
- std::string scan_node_type = "VOdbcScanNode");
- ~VOdbcScanNode() = default;
-
- // initialize odbc_scanner, and create text_converter.
- virtual Status prepare(RuntimeState* state) override;
-
- // Start ODBC scan using odbc_scanner.
- virtual Status open(RuntimeState* state) override;
-
- Status get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) override {
- return Status::NotSupported("Not Implemented VOdbcScanNode Node::get_next scalar");
- }
- Status get_next(RuntimeState* state, Block* block, bool* eos) override;
-
- // Close the odbc_scanner, and report errors.
- virtual Status close(RuntimeState* state) override;
- // No use
- virtual Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) override;
- const TupleDescriptor* get_tuple_desc() { return _tuple_desc; }
- TextConverter* get_text_converter() { return _text_converter.get(); }
- ODBCConnector* get_odbc_scanner() { return _odbc_scanner.get(); }
- const std::string& get_scan_node_type() { return _scan_node_type; }
-
- bool is_init() { return _is_init; }
-
-protected:
- // Write debug string of this into out.
- virtual void debug_string(int indentation_level, std::stringstream* out) const override;
-
-private:
- bool _is_init;
-
- std::string _scan_node_type;
-
- // Name of Odbc table
- std::string _table_name;
-
- std::string _connect_string;
-
- std::string _query_string;
- // Tuple id resolved in prepare() to set _tuple_desc;
- TupleId _tuple_id;
-
- // Descriptor of tuples read from ODBC table.
- const TupleDescriptor* _tuple_desc;
- // Tuple index in tuple row.
- int _slot_num;
- // Pool for allocating tuple data, including all varying-length slots.
- std::unique_ptr<MemPool> _tuple_pool;
-
- // Scanner of ODBC.
- std::unique_ptr<ODBCConnector> _odbc_scanner;
- ODBCConnectorParam _odbc_param;
- // Helper class for converting text to other types;
- std::unique_ptr<TextConverter> _text_converter;
-};
-} // namespace vectorized
-} // namespace doris
diff --git a/be/src/vec/exec/volap_scan_node.cpp b/be/src/vec/exec/volap_scan_node.cpp
deleted file mode 100644
index 19ab63ae37..0000000000
--- a/be/src/vec/exec/volap_scan_node.cpp
+++ /dev/null
@@ -1,1843 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include "vec/exec/volap_scan_node.h"
-
-#include "common/resource_tls.h"
-#include "exec/scan_node.h"
-#include "gen_cpp/PlanNodes_types.h"
-#include "olap/storage_engine.h"
-#include "runtime/descriptors.h"
-#include "runtime/exec_env.h"
-#include "runtime/large_int_value.h"
-#include "runtime/runtime_filter_mgr.h"
-#include "util/priority_thread_pool.hpp"
-#include "util/to_string.h"
-#include "vec/core/block.h"
-#include "vec/data_types/data_type_decimal.h"
-#include "vec/exec/volap_scanner.h"
-#include "vec/exprs/vbloom_predicate.h"
-#include "vec/exprs/vcompound_pred.h"
-#include "vec/exprs/vexpr.h"
-#include "vec/exprs/vruntimefilter_wrapper.h"
-#include "vec/functions/in.h"
-
-namespace doris::vectorized {
-using doris::operator<<;
-
-#define RETURN_IF_PUSH_DOWN(stmt) \
- if (!push_down) { \
- stmt; \
- } else { \
- return; \
- }
-
-VOlapScanNode::VOlapScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs)
- : ScanNode(pool, tnode, descs),
- _tuple_id(tnode.olap_scan_node.tuple_id),
- _olap_scan_node(tnode.olap_scan_node),
- _tuple_desc(nullptr),
- _tuple_idx(0),
- _eos(false),
- _max_materialized_row_batches(config::doris_scanner_queue_size),
- _start(false),
- _scanner_done(false),
- _transfer_done(false),
- _status(Status::OK()),
- _resource_info(nullptr),
- _buffered_bytes(0),
- _eval_conjuncts_fn(nullptr),
- _runtime_filter_descs(tnode.runtime_filters),
- _max_materialized_blocks(config::doris_scanner_queue_size) {
- _materialized_blocks.reserve(_max_materialized_blocks);
- _free_blocks.reserve(_max_materialized_blocks);
- // if sort_info is set, push _limit to each olap scanner
- if (_olap_scan_node.__isset.sort_info && _olap_scan_node.__isset.sort_limit) {
- _limit_per_scanner = _olap_scan_node.sort_limit;
- }
-}
-
-Status VOlapScanNode::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(ExecNode::init(tnode, state));
-
- const TQueryOptions& query_options = state->query_options();
- if (query_options.__isset.max_scan_key_num) {
- _max_scan_key_num = query_options.max_scan_key_num;
- } else {
- _max_scan_key_num = config::doris_max_scan_key_num;
- }
-
- if (query_options.__isset.max_pushdown_conditions_per_column) {
- _max_pushdown_conditions_per_column = query_options.max_pushdown_conditions_per_column;
- } else {
- _max_pushdown_conditions_per_column = config::max_pushdown_conditions_per_column;
- }
-
- _max_scanner_queue_size_bytes = query_options.mem_limit / 20; //TODO: session variable percent
-
- /// TODO: could one filter used in the different scan_node ?
- int filter_size = _runtime_filter_descs.size();
- _runtime_filter_ctxs.resize(filter_size);
- _runtime_filter_ready_flag.resize(filter_size);
- for (int i = 0; i < filter_size; ++i) {
- IRuntimeFilter* runtime_filter = nullptr;
- const auto& filter_desc = _runtime_filter_descs[i];
- RETURN_IF_ERROR(state->runtime_filter_mgr()->regist_filter(
- RuntimeFilterRole::CONSUMER, filter_desc, state->query_options(), id()));
- RETURN_IF_ERROR(state->runtime_filter_mgr()->get_consume_filter(filter_desc.filter_id,
- &runtime_filter));
-
- _runtime_filter_ctxs[i].runtimefilter = runtime_filter;
- _runtime_filter_ready_flag[i] = false;
- }
-
- return Status::OK();
-}
-
-void VOlapScanNode::init_scan_profile() {
- std::string scanner_profile_name = "VOlapScanner";
- if (_olap_scan_node.__isset.table_name) {
- scanner_profile_name = fmt::format("VOlapScanner({0})", _olap_scan_node.table_name);
- }
- _scanner_profile.reset(new RuntimeProfile(scanner_profile_name));
- runtime_profile()->add_child(_scanner_profile.get(), true, nullptr);
-
- _segment_profile.reset(new RuntimeProfile("SegmentIterator"));
- _scanner_profile->add_child(_segment_profile.get(), true, nullptr);
-}
-
-void VOlapScanNode::_init_counter(RuntimeState* state) {
- ADD_TIMER(_scanner_profile, "ShowHintsTime_V1");
-
- _reader_init_timer = ADD_TIMER(_scanner_profile, "ReaderInitTime");
- _read_compressed_counter = ADD_COUNTER(_segment_profile, "CompressedBytesRead", TUnit::BYTES);
- _read_uncompressed_counter =
- ADD_COUNTER(_segment_profile, "UncompressedBytesRead", TUnit::BYTES);
- _block_load_timer = ADD_TIMER(_segment_profile, "BlockLoadTime");
- _block_load_counter = ADD_COUNTER(_segment_profile, "BlocksLoad", TUnit::UNIT);
- _block_fetch_timer = ADD_TIMER(_scanner_profile, "BlockFetchTime");
- _raw_rows_counter = ADD_COUNTER(_segment_profile, "RawRowsRead", TUnit::UNIT);
- _block_convert_timer = ADD_TIMER(_scanner_profile, "BlockConvertTime");
- // Will be delete after non-vectorized code is removed
- _block_seek_timer = ADD_TIMER(_segment_profile, "BlockSeekTime");
- _block_seek_counter = ADD_COUNTER(_segment_profile, "BlockSeekCount", TUnit::UNIT);
- _block_init_timer = ADD_TIMER(_segment_profile, "BlockInitTime");
- _block_init_seek_timer = ADD_TIMER(_segment_profile, "BlockInitSeekTime");
- _block_init_seek_counter = ADD_COUNTER(_segment_profile, "BlockInitSeekCount", TUnit::UNIT);
-
- _rows_vec_cond_counter = ADD_COUNTER(_segment_profile, "RowsVectorPredFiltered", TUnit::UNIT);
- _vec_cond_timer = ADD_TIMER(_segment_profile, "VectorPredEvalTime");
- _short_cond_timer = ADD_TIMER(_segment_profile, "ShortPredEvalTime");
- _first_read_timer = ADD_TIMER(_segment_profile, "FirstReadTime");
- _first_read_seek_timer = ADD_TIMER(_segment_profile, "FirstReadSeekTime");
- _first_read_seek_counter = ADD_COUNTER(_segment_profile, "FirstReadSeekCount", TUnit::UNIT);
-
- _lazy_read_timer = ADD_TIMER(_segment_profile, "LazyReadTime");
- _lazy_read_seek_timer = ADD_TIMER(_segment_profile, "LazyReadSeekTime");
- _lazy_read_seek_counter = ADD_COUNTER(_segment_profile, "LazyReadSeekCount", TUnit::UNIT);
-
- _output_col_timer = ADD_TIMER(_segment_profile, "OutputColumnTime");
-
- _stats_filtered_counter = ADD_COUNTER(_segment_profile, "RowsStatsFiltered", TUnit::UNIT);
- _bf_filtered_counter = ADD_COUNTER(_segment_profile, "RowsBloomFilterFiltered", TUnit::UNIT);
- _del_filtered_counter = ADD_COUNTER(_scanner_profile, "RowsDelFiltered", TUnit::UNIT);
- _conditions_filtered_counter =
- ADD_COUNTER(_segment_profile, "RowsConditionsFiltered", TUnit::UNIT);
- _key_range_filtered_counter =
- ADD_COUNTER(_segment_profile, "RowsKeyRangeFiltered", TUnit::UNIT);
-
- _io_timer = ADD_TIMER(_segment_profile, "IOTimer");
- _decompressor_timer = ADD_TIMER(_segment_profile, "DecompressorTimer");
- _index_load_timer = ADD_TIMER(_segment_profile, "IndexLoadTime_V1");
-
- _scan_timer = ADD_TIMER(_scanner_profile, "ScanTime");
- _scan_cpu_timer = ADD_TIMER(_scanner_profile, "ScanCpuTime");
-
- _total_pages_num_counter = ADD_COUNTER(_segment_profile, "TotalPagesNum", TUnit::UNIT);
- _cached_pages_num_counter = ADD_COUNTER(_segment_profile, "CachedPagesNum", TUnit::UNIT);
-
- _bitmap_index_filter_counter =
- ADD_COUNTER(_segment_profile, "RowsBitmapIndexFiltered", TUnit::UNIT);
- _bitmap_index_filter_timer = ADD_TIMER(_segment_profile, "BitmapIndexFilterTimer");
-
- _num_scanners = ADD_COUNTER(_runtime_profile, "NumScanners", TUnit::UNIT);
-
- _filtered_segment_counter = ADD_COUNTER(_segment_profile, "NumSegmentFiltered", TUnit::UNIT);
- _total_segment_counter = ADD_COUNTER(_segment_profile, "NumSegmentTotal", TUnit::UNIT);
-
- // time of transfer thread to wait for row batch from scan thread
- _scanner_wait_batch_timer = ADD_TIMER(_runtime_profile, "ScannerBatchWaitTime");
- // time of scan thread to wait for worker thread of the thread pool
- _scanner_wait_worker_timer = ADD_TIMER(_runtime_profile, "ScannerWorkerWaitTime");
-
- // time of node to wait for batch/block queue
- _olap_wait_batch_queue_timer = ADD_TIMER(_runtime_profile, "BatchQueueWaitTime");
-
- // for the purpose of debugging or profiling
- for (int i = 0; i < GENERAL_DEBUG_COUNT; ++i) {
- char name[64];
- snprintf(name, sizeof(name), "GeneralDebugTimer%d", i);
- _general_debug_timer[i] = ADD_TIMER(_segment_profile, name);
- }
-}
-
-Status VOlapScanNode::prepare(RuntimeState* state) {
- init_scan_profile();
- RETURN_IF_ERROR(ScanNode::prepare(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
- // create scanner profile
- // create timer
- _tablet_counter = ADD_COUNTER(runtime_profile(), "TabletCount ", TUnit::UNIT);
- _scanner_sched_counter = ADD_COUNTER(runtime_profile(), "ScannerSchedCount ", TUnit::UNIT);
-
- _rows_pushed_cond_filtered_counter =
- ADD_COUNTER(_scanner_profile, "RowsPushedCondFiltered", TUnit::UNIT);
- _init_counter(state);
- _tuple_desc = state->desc_tbl().get_tuple_descriptor(_tuple_id);
-
- if (_tuple_desc == nullptr) {
- // TODO: make sure we print all available diagnostic output to our error log
- return Status::InternalError("Failed to get tuple descriptor.");
- }
-
- const std::vector<SlotDescriptor*>& slots = _tuple_desc->slots();
-
- for (int i = 0; i < slots.size(); ++i) {
- if (!slots[i]->is_materialized()) {
- continue;
- }
-
- if (slots[i]->type().is_collection_type()) {
- _collection_slots.push_back(slots[i]);
- }
-
- if (slots[i]->type().is_string_type()) {
- _string_slots.push_back(slots[i]);
- }
- }
-
- _runtime_state = state;
- for (size_t i = 0; i < _runtime_filter_descs.size(); ++i) {
- _runtime_filter_ctxs[i].runtimefilter->init_profile(_runtime_profile.get());
- }
- return Status::OK();
-}
-
-Status VOlapScanNode::open(RuntimeState* state) {
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VOlapScanNode::open");
- VLOG_CRITICAL << "VOlapScanNode::Open";
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- RETURN_IF_CANCELLED(state);
- RETURN_IF_ERROR(ExecNode::open(state));
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
-
- _resource_info = ResourceTls::get_resource_tls();
-
- // acquire runtime filter
- _runtime_filter_ctxs.resize(_runtime_filter_descs.size());
-
- std::vector<VExpr*> vexprs;
- for (size_t i = 0; i < _runtime_filter_descs.size(); ++i) {
- IRuntimeFilter* runtime_filter = _runtime_filter_ctxs[i].runtimefilter;
- // If all targets are local, scan node will use hash node's runtime filter, and we don't
- // need to allocate memory again
- if (runtime_filter->has_remote_target()) {
- if (auto bf = runtime_filter->get_bloomfilter()) {
- RETURN_IF_ERROR(bf->init_with_fixed_length());
- }
- }
- bool ready = runtime_filter->is_ready();
- if (!ready) {
- ready = runtime_filter->await();
- }
- if (ready) {
- RETURN_IF_ERROR(runtime_filter->get_push_expr_ctxs(&vexprs));
- _runtime_filter_ctxs[i].apply_mark = true;
- _runtime_filter_ctxs[i].runtimefilter = runtime_filter;
- }
- }
- RETURN_IF_ERROR(_append_rf_into_conjuncts(state, vexprs));
-
- return Status::OK();
-}
-
-void VOlapScanNode::transfer_thread(RuntimeState* state) {
- // scanner open pushdown to scanThread
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VOlapScanNode::transfer_thread");
- SCOPED_ATTACH_TASK(state);
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker_shared());
- Status status = Status::OK();
-
- if (_vconjunct_ctx_ptr) {
- for (auto scanner : _volap_scanners) {
- status = (*_vconjunct_ctx_ptr)->clone(state, scanner->vconjunct_ctx_ptr());
- if (!status.ok()) {
- std::lock_guard<SpinLock> guard(_status_mutex);
- _status = status;
- break;
- }
- }
- }
-
- /*********************************
- * 优先级调度基本策略:
- * 1. 通过查询拆分的Range个数来确定初始nice值
- * Range个数越多,越倾向于认定为大查询,nice值越小
- * 2. 通过查询累计读取的数据量来调整nice值
- * 读取的数据越多,越倾向于认定为大查询,nice值越小
- * 3. 通过nice值来判断查询的优先级
- * nice值越大的,越优先获得的查询资源
- * 4. 定期提高队列内残留任务的优先级,避免大查询完全饿死
- *********************************/
- _total_assign_num = 0;
- _nice = 18 + std::max(0, 2 - (int)_volap_scanners.size() / 5);
-
- auto doris_scanner_row_num =
- _limit == -1 ? config::doris_scanner_row_num
- : std::min(static_cast<int64_t>(config::doris_scanner_row_num), _limit);
- _block_size = _limit == -1 ? state->batch_size()
- : std::min(static_cast<int64_t>(state->batch_size()), _limit);
- auto block_per_scanner = (doris_scanner_row_num + (_block_size - 1)) / _block_size;
- auto pre_block_count =
- std::min(_volap_scanners.size(),
- static_cast<size_t>(config::doris_scanner_thread_pool_thread_num)) *
- block_per_scanner;
-
- for (int i = 0; i < pre_block_count; ++i) {
- auto block = new Block(_tuple_desc->slots(), _block_size);
- _free_blocks.emplace_back(block);
- _buffered_bytes += block->allocated_bytes();
- }
-
- // read from scanner
- while (LIKELY(status.ok())) {
- int assigned_thread_num = _start_scanner_thread_task(state, block_per_scanner);
-
- std::vector<Block*> blocks;
- {
- // 1 scanner idle task not empty, assign new scanner task
- std::unique_lock<std::mutex> l(_scan_blocks_lock);
-
- // scanner_row_num = 16k
- // 16k * 10 * 12 * 8 = 15M(>2s) --> nice=10
- // 16k * 20 * 22 * 8 = 55M(>6s) --> nice=0
- while (_nice > 0 && _total_assign_num > (22 - _nice) * (20 - _nice) * 6) {
- --_nice;
- }
-
- // 2 wait when all scanner are running & no result in queue
- while (UNLIKELY(_running_thread == assigned_thread_num && _scan_blocks.empty() &&
- !_scanner_done)) {
- SCOPED_TIMER(_scanner_wait_batch_timer);
- _scan_block_added_cv.wait(l);
- }
-
- // 3 transfer result block when queue is not empty
- if (LIKELY(!_scan_blocks.empty())) {
- blocks.swap(_scan_blocks);
- for (auto b : blocks) {
- _scan_row_batches_bytes -= b->allocated_bytes();
- }
- // delete scan_block if transfer thread should be stopped
- // because scan_block wouldn't be useful anymore
- if (UNLIKELY(_transfer_done)) {
- std::for_each(blocks.begin(), blocks.end(), std::default_delete<Block>());
- blocks.clear();
- }
- } else {
- if (_scanner_done) {
- // We should close eof scanners before transfer done, otherwise,
- // they are closed until scannode is closed. Because plan is closed
- // after the plan is finished, so query profile would leak stats from
- // scanners closed by scannode::close.
- while (!_volap_scanners.empty()) {
- auto scanner = _volap_scanners.front();
- _volap_scanners.pop_front();
- DCHECK(scanner->need_to_close());
- scanner->close(state);
- }
- break;
- }
- }
- }
-
- if (!blocks.empty()) {
- _add_blocks(blocks);
- }
- }
-
- VLOG_CRITICAL << "TransferThread finish.";
- {
- std::unique_lock<std::mutex> l(_blocks_lock);
- _transfer_done = true;
- }
- _block_added_cv.notify_all();
- {
- std::unique_lock<std::mutex> l(_scan_blocks_lock);
- _scan_thread_exit_cv.wait(l, [this] { return _running_thread == 0; });
- }
- VLOG_CRITICAL << "Scanner threads have been exited. TransferThread exit.";
-}
-
-void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
- SCOPED_ATTACH_TASK(_runtime_state->scanner_mem_tracker(),
- ThreadContext::query_to_task_type(_runtime_state->query_type()),
- print_id(_runtime_state->query_id()),
- _runtime_state->fragment_instance_id());
- Thread::set_self_name("volap_scanner");
- int64_t wait_time = scanner->update_wait_worker_timer();
- // Do not use ScopedTimer. There is no guarantee that, the counter
- // (_scan_cpu_timer, the class member) is not destroyed after `_running_thread==0`.
- ThreadCpuStopWatch cpu_watch;
- cpu_watch.start();
- Status status = Status::OK();
- bool eos = false;
- RuntimeState* state = scanner->runtime_state();
- DCHECK(nullptr != state);
- if (!scanner->is_open()) {
- status = scanner->open();
- if (!status.ok()) {
- std::lock_guard<SpinLock> guard(_status_mutex);
- _status = status;
- eos = true;
- }
- scanner->set_opened();
- }
-
- std::vector<VExpr*> vexprs;
- auto& scanner_filter_apply_marks = *scanner->mutable_runtime_filter_marks();
- DCHECK(scanner_filter_apply_marks.size() == _runtime_filter_descs.size());
- for (size_t i = 0; i < scanner_filter_apply_marks.size(); i++) {
- if (!scanner_filter_apply_marks[i] && !_runtime_filter_ctxs[i].apply_mark) {
- /// When runtime filters are ready during running, we should use them to filter data
- /// in VOlapScanner.
- /// New arrival rf will be processed as below:
- /// 1. convert these runtime filters to vectorized expressions
- /// 2. if this is the first scanner thread to receive this rf, construct a new
- /// VExprContext and update `_vconjunct_ctx_ptr` in scan node. Notice that we use
- /// `_runtime_filter_ready_flag` to ensure `_vconjunct_ctx_ptr` will be updated only
- /// once after any runtime_filters are ready.
- /// 3. finally, just copy this new VExprContext to scanner and use it to filter data.
- IRuntimeFilter* runtime_filter = _runtime_filter_ctxs[i].runtimefilter;
- DCHECK(runtime_filter != nullptr);
- bool ready = runtime_filter->is_ready();
- if (ready) {
- runtime_filter->get_prepared_vexprs(&vexprs, _row_descriptor);
- scanner_filter_apply_marks[i] = true;
- if (!_runtime_filter_ready_flag[i] && !vexprs.empty()) {
- std::lock_guard<std::shared_mutex> l(_rf_lock);
- if (!_runtime_filter_ready_flag[i]) {
- // Use all conjuncts and new arrival runtime filters to construct a new
- // expression tree here.
- _append_rf_into_conjuncts(state, vexprs);
- _runtime_filter_ready_flag[i] = true;
- }
- }
- }
- }
- }
-
- if (!vexprs.empty()) {
- if (*scanner->vconjunct_ctx_ptr()) {
- scanner->discard_conjuncts();
- }
- {
- std::shared_lock<std::shared_mutex> l(_rf_lock);
- WARN_IF_ERROR((*_vconjunct_ctx_ptr)->clone(state, scanner->vconjunct_ctx_ptr()),
- "Something wrong for runtime filters: ");
- }
- }
-
- std::vector<Block*> blocks;
-
- // Because we use thread pool to scan data from storage. One scanner can't
- // use this thread too long, this can starve other query's scanner. So, we
- // need yield this thread when we do enough work. However, OlapStorage read
- // data in pre-aggregate mode, then we can't use storage returned data to
- // judge if we need to yield. So we record all raw data read in this round
- // scan, if this exceed row number or bytes threshold, we yield this thread.
- int64_t raw_rows_read = scanner->raw_rows_read();
- int64_t raw_rows_threshold = raw_rows_read + config::doris_scanner_row_num;
- int64_t raw_bytes_read = 0;
- int64_t raw_bytes_threshold = config::doris_scanner_row_bytes;
- bool get_free_block = true;
- int num_rows_in_block = 0;
-
- // Has to wait at least one full block, or it will cause a lot of schedule task in priority
- // queue, it will affect query latency and query concurrency for example ssb 3.3.
- while (!eos && raw_bytes_read < raw_bytes_threshold &&
- ((raw_rows_read < raw_rows_threshold && get_free_block) ||
- num_rows_in_block < _runtime_state->batch_size())) {
- if (UNLIKELY(_transfer_done)) {
- eos = true;
- status = Status::Cancelled(
- "Scan thread cancelled, cause query done, maybe reach limit.");
- break;
- }
-
- auto block = _alloc_block(get_free_block);
- status = scanner->get_block(_runtime_state, block, &eos);
- VLOG_ROW << "VOlapScanNode input rows: " << block->rows();
- if (!status.ok()) {
- LOG(WARNING) << "Scan thread read VOlapScanner failed: " << status.to_string();
- // Add block ptr in blocks, prevent mem leak in read failed
- blocks.push_back(block);
- eos = true;
- break;
- }
-
- raw_bytes_read += block->bytes();
- num_rows_in_block += block->rows();
- // 4. if status not ok, change status_.
- if (UNLIKELY(block->rows() == 0)) {
- std::lock_guard<std::mutex> l(_free_blocks_lock);
- _free_blocks.emplace_back(block);
- } else {
- if (!blocks.empty() &&
- blocks.back()->rows() + block->rows() <= _runtime_state->batch_size()) {
- MutableBlock(blocks.back()).merge(*block);
- block->clear_column_data();
- std::lock_guard<std::mutex> l(_free_blocks_lock);
- _free_blocks.emplace_back(block);
- } else {
- blocks.push_back(block);
- }
- }
- raw_rows_read = scanner->raw_rows_read();
- }
-
- {
- // if we failed, check status.
- if (UNLIKELY(!status.ok())) {
- _transfer_done = true;
- std::lock_guard<SpinLock> guard(_status_mutex);
- if (LIKELY(_status.ok())) {
- _status = status;
- }
- }
-
- bool global_status_ok = false;
- {
- std::lock_guard<SpinLock> guard(_status_mutex);
- global_status_ok = _status.ok();
- }
- if (UNLIKELY(!global_status_ok)) {
- eos = true;
- std::for_each(blocks.begin(), blocks.end(), std::default_delete<Block>());
- } else {
- std::lock_guard<std::mutex> l(_scan_blocks_lock);
- _scan_blocks.insert(_scan_blocks.end(), blocks.begin(), blocks.end());
- for (auto b : blocks) {
- _scan_row_batches_bytes += b->allocated_bytes();
- }
- }
- // If eos is true, we will process out of this lock block.
- if (eos) {
- scanner->mark_to_need_to_close();
- }
- std::lock_guard<std::mutex> l(_volap_scanners_lock);
- _volap_scanners.push_front(scanner);
- }
- if (eos) {
- std::lock_guard<std::mutex> l(_scan_blocks_lock);
- _progress.update(1);
- if (_progress.done()) {
- // this is the right out
- _scanner_done = true;
- }
- }
- _scan_cpu_timer->update(cpu_watch.elapsed_time());
- _scanner_wait_worker_timer->update(wait_time);
-
- std::unique_lock<std::mutex> l(_scan_blocks_lock);
- _running_thread--;
-
- // The transfer thead will wait for `_running_thread==0`, to make sure all scanner threads won't access class members.
- // Do not access class members after this code.
- _scan_block_added_cv.notify_one();
- _scan_thread_exit_cv.notify_one();
-}
-
-Status VOlapScanNode::_add_blocks(std::vector<Block*>& block) {
- {
- std::unique_lock<std::mutex> l(_blocks_lock);
-
- // check queue limit for both block queue size and bytes
- while (UNLIKELY((_materialized_blocks.size() >= _max_materialized_blocks ||
- _materialized_row_batches_bytes >= _max_scanner_queue_size_bytes / 2) &&
- !_transfer_done)) {
- _block_consumed_cv.wait(l);
- }
-
- VLOG_CRITICAL << "Push block to materialized_blocks";
- _materialized_blocks.insert(_materialized_blocks.end(), block.cbegin(), block.cend());
- for (auto b : block) {
- _materialized_row_batches_bytes += b->allocated_bytes();
- }
- }
- // remove one block, notify main thread
- _block_added_cv.notify_one();
- return Status::OK();
-}
-
-Status VOlapScanNode::normalize_conjuncts() {
- std::vector<SlotDescriptor*> slots = _tuple_desc->slots();
-
- for (int slot_idx = 0; slot_idx < slots.size(); ++slot_idx) {
- switch (slots[slot_idx]->type().type) {
-#define M(NAME) \
- case TYPE_##NAME: { \
- ColumnValueRange<TYPE_##NAME> range(slots[slot_idx]->col_name(), \
- slots[slot_idx]->type().precision, \
- slots[slot_idx]->type().scale); \
- _id_to_slot_column_value_range[slots[slot_idx]->id()] = \
- std::pair {slots[slot_idx], range}; \
- break; \
- }
-#define APPLY_FOR_PRIMITIVE_TYPE(M) \
- M(TINYINT) \
- M(SMALLINT) \
- M(INT) \
- M(BIGINT) \
- M(LARGEINT) \
- M(CHAR) \
- M(DATE) \
- M(DATETIME) \
- M(DATEV2) \
- M(DATETIMEV2) \
- M(VARCHAR) \
- M(STRING) \
- M(HLL) \
- M(DECIMAL32) \
- M(DECIMAL64) \
- M(DECIMAL128) \
- M(DECIMALV2) \
- M(BOOLEAN)
- APPLY_FOR_PRIMITIVE_TYPE(M)
-#undef M
- default: {
- VLOG_CRITICAL << "Unsupported Normalize Slot [ColName=" << slots[slot_idx]->col_name()
- << "]";
- break;
- }
- }
- }
- if (_vconjunct_ctx_ptr) {
- if ((*_vconjunct_ctx_ptr)->root()) {
- VExpr* new_root = _normalize_predicate(_runtime_state, (*_vconjunct_ctx_ptr)->root());
- if (new_root) {
- (*_vconjunct_ctx_ptr)->set_root(new_root);
- } else {
- (*(_vconjunct_ctx_ptr.get()))->mark_as_stale();
- _stale_vexpr_ctxs.push_back(std::move(_vconjunct_ctx_ptr));
- _vconjunct_ctx_ptr.reset(nullptr);
- }
- }
- }
- for (auto& it : _id_to_slot_column_value_range) {
- std::visit(
- [&](auto&& range) {
- if (range.is_empty_value_range()) {
- _eos = true;
- }
- },
- it.second.second);
- _column_value_ranges[it.second.first->col_name()] = it.second.second;
- }
-
- return Status::OK();
-}
-
-static std::string olap_filter_to_string(const doris::TCondition& condition) {
- auto op_name = condition.condition_op;
- if (condition.condition_op == "*=") {
- op_name = "IN";
- } else if (condition.condition_op == "!*=") {
- op_name = "NOT IN";
- }
- return fmt::format("{{{} {} {}}}", condition.column_name, op_name,
- to_string(condition.condition_values));
-}
-
-static std::string olap_filters_to_string(const std::vector<doris::TCondition>& filters) {
- std::string filters_string;
- filters_string += "[";
- for (auto it = filters.cbegin(); it != filters.cend(); it++) {
- if (it != filters.cbegin()) {
- filters_string += ",";
- }
- filters_string += olap_filter_to_string(*it);
- }
- filters_string += "]";
- return filters_string;
-}
-
-Status VOlapScanNode::build_key_ranges_and_filters() {
- const std::vector<std::string>& column_names = _olap_scan_node.key_column_name;
- const std::vector<TPrimitiveType::type>& column_types = _olap_scan_node.key_column_type;
- DCHECK(column_types.size() == column_names.size());
-
- // 1. construct scan key except last olap engine short key
- _scan_keys.set_is_convertible(limit() == -1);
-
- // we use `exact_range` to identify a key range is an exact range or not when we convert
- // it to `_scan_keys`. If `exact_range` is true, we can just discard it from `_olap_filter`.
- bool exact_range = true;
- for (int column_index = 0; column_index < column_names.size() && !_scan_keys.has_range_value();
- ++column_index) {
- auto iter = _column_value_ranges.find(column_names[column_index]);
- if (_column_value_ranges.end() == iter) {
- break;
- }
-
- RETURN_IF_ERROR(std::visit(
- [&](auto&& range) {
- // make a copy or range and pass to extend_scan_key, keep the range unchanged
- // because extend_scan_key method may change the first parameter.
- // but the original range may be converted to olap filters, if it's not a exact_range.
- auto temp_range = range;
- RETURN_IF_ERROR(_scan_keys.extend_scan_key(temp_range, _max_scan_key_num,
- &exact_range));
- if (exact_range) {
- _column_value_ranges.erase(iter->first);
- }
- return Status::OK();
- },
- iter->second));
- }
-
- for (auto& iter : _column_value_ranges) {
- std::vector<TCondition> filters;
- std::visit([&](auto&& range) { range.to_olap_filter(filters); }, iter.second);
-
- for (const auto& filter : filters) {
- _olap_filter.push_back(std::move(filter));
- }
- }
-
- _runtime_profile->add_info_string("PushdownPredicate", olap_filters_to_string(_olap_filter));
-
- _runtime_profile->add_info_string("KeyRanges", _scan_keys.debug_string());
-
- VLOG_CRITICAL << _scan_keys.debug_string();
-
- return Status::OK();
-}
-
-Status VOlapScanNode::start_scan(RuntimeState* state) {
- RETURN_IF_CANCELLED(state);
-
- VLOG_CRITICAL << "NormalizeConjuncts";
- RETURN_IF_ERROR(normalize_conjuncts());
-
- if (_eos) {
- return Status::OK();
- }
-
- VLOG_CRITICAL << "BuildKeyRangesAndFilters";
- RETURN_IF_ERROR(build_key_ranges_and_filters());
-
- VLOG_CRITICAL << "StartScanThread";
- RETURN_IF_ERROR(start_scan_thread(state));
-
- return Status::OK();
-}
-
-static bool ignore_cast(SlotDescriptor* slot, VExpr* expr) {
- if (slot->type().is_date_type() && expr->type().is_date_type()) {
- return true;
- }
- if (slot->type().is_string_type() && expr->type().is_string_type()) {
- return true;
- }
- return false;
-}
-
-template <bool IsFixed, PrimitiveType PrimitiveType, typename ChangeFixedValueRangeFunc>
-Status VOlapScanNode::change_value_range(ColumnValueRange<PrimitiveType>& temp_range, void* value,
- const ChangeFixedValueRangeFunc& func,
- const std::string& fn_name, int slot_ref_child) {
- if constexpr (PrimitiveType == TYPE_DATE) {
- DateTimeValue date_value;
- reinterpret_cast<VecDateTimeValue*>(value)->convert_vec_dt_to_dt(&date_value);
- if constexpr (IsFixed) {
- if (!date_value.check_loss_accuracy_cast_to_date()) {
- func(temp_range,
- reinterpret_cast<typename PrimitiveTypeTraits<PrimitiveType>::CppType*>(
- &date_value));
- }
- } else {
- if (date_value.check_loss_accuracy_cast_to_date()) {
- if (fn_name == "lt" || fn_name == "ge") {
- ++date_value;
- }
- }
- func(temp_range, to_olap_filter_type(fn_name, slot_ref_child),
- reinterpret_cast<typename PrimitiveTypeTraits<PrimitiveType>::CppType*>(
- &date_value));
- }
- } else if constexpr (PrimitiveType == TYPE_DATETIME) {
- DateTimeValue date_value;
- reinterpret_cast<VecDateTimeValue*>(value)->convert_vec_dt_to_dt(&date_value);
- if constexpr (IsFixed) {
- func(temp_range,
- reinterpret_cast<typename PrimitiveTypeTraits<PrimitiveType>::CppType*>(
- &date_value));
- } else {
- func(temp_range, to_olap_filter_type(fn_name, slot_ref_child),
- reinterpret_cast<typename PrimitiveTypeTraits<PrimitiveType>::CppType*>(
- reinterpret_cast<char*>(&date_value)));
- }
- } else if constexpr ((PrimitiveType == TYPE_DECIMALV2) || (PrimitiveType == TYPE_CHAR) ||
- (PrimitiveType == TYPE_VARCHAR) || (PrimitiveType == TYPE_HLL) ||
- (PrimitiveType == TYPE_DATETIMEV2) || (PrimitiveType == TYPE_TINYINT) ||
- (PrimitiveType == TYPE_SMALLINT) || (PrimitiveType == TYPE_INT) ||
- (PrimitiveType == TYPE_BIGINT) || (PrimitiveType == TYPE_LARGEINT) ||
- (PrimitiveType == TYPE_DECIMAL32) || (PrimitiveType == TYPE_DECIMAL64) ||
- (PrimitiveType == TYPE_DECIMAL128) || (PrimitiveType == TYPE_STRING) ||
- (PrimitiveType == TYPE_BOOLEAN) || (PrimitiveType == TYPE_DATEV2)) {
- if constexpr (IsFixed) {
- func(temp_range,
- reinterpret_cast<typename PrimitiveTypeTraits<PrimitiveType>::CppType*>(value));
- } else {
- func(temp_range, to_olap_filter_type(fn_name, slot_ref_child),
- reinterpret_cast<typename PrimitiveTypeTraits<PrimitiveType>::CppType*>(value));
- }
- } else {
- static_assert(always_false_v<PrimitiveType>);
- }
-
- return Status::OK();
-}
-
-bool VOlapScanNode::is_key_column(const std::string& key_name) {
- // all column in dup_keys table or unique_keys with merge on write table olap scan node threat
- // as key column
- if (_olap_scan_node.keyType == TKeysType::DUP_KEYS ||
- (_olap_scan_node.keyType == TKeysType::UNIQUE_KEYS &&
- _olap_scan_node.__isset.enable_unique_key_merge_on_write &&
- _olap_scan_node.enable_unique_key_merge_on_write)) {
- return true;
- }
-
- auto res = std::find(_olap_scan_node.key_column_name.begin(),
- _olap_scan_node.key_column_name.end(), key_name);
- return res != _olap_scan_node.key_column_name.end();
-}
-
-Status VOlapScanNode::start_scan_thread(RuntimeState* state) {
- if (_scan_ranges.empty()) {
- _transfer_done = true;
- return Status::OK();
- }
-
- // ranges constructed from scan keys
- std::vector<std::unique_ptr<OlapScanRange>> cond_ranges;
- RETURN_IF_ERROR(_scan_keys.get_key_range(&cond_ranges));
- // if we can't get ranges from conditions, we give it a total range
- if (cond_ranges.empty()) {
- cond_ranges.emplace_back(new OlapScanRange());
- }
- bool need_split = true;
- // If we have ranges more than 64, there is no need to call
- // ShowHint to split ranges
- if (limit() != -1 || cond_ranges.size() > 64) {
- need_split = false;
- }
- int scanners_per_tablet = std::max(1, 64 / (int)_scan_ranges.size());
-
- std::unordered_set<std::string> disk_set;
- for (auto& scan_range : _scan_ranges) {
- auto tablet_id = scan_range->tablet_id;
- std::string err;
- TabletSharedPtr tablet =
- StorageEngine::instance()->tablet_manager()->get_tablet(tablet_id, true, &err);
- if (tablet == nullptr) {
- std::stringstream ss;
- ss << "failed to get tablet: " << tablet_id << ", reason: " << err;
- LOG(WARNING) << ss.str();
- return Status::InternalError(ss.str());
- }
-
- std::vector<std::unique_ptr<OlapScanRange>>* ranges = &cond_ranges;
- std::vector<std::unique_ptr<OlapScanRange>> split_ranges;
- if (need_split && !tablet->all_beta()) {
- auto st = get_hints(tablet, *scan_range, config::doris_scan_range_row_count,
- _scan_keys.begin_include(), _scan_keys.end_include(), cond_ranges,
- &split_ranges, _runtime_profile.get());
- if (st.ok()) {
- ranges = &split_ranges;
- }
- }
- int size_based_scanners_per_tablet = 1;
-
- if (config::doris_scan_range_max_mb > 0) {
- size_based_scanners_per_tablet = std::max(
- 1, (int)(tablet->tablet_footprint() / (config::doris_scan_range_max_mb << 20)));
- }
-
- int ranges_per_scanner =
- std::max(1, (int)ranges->size() /
- std::min(scanners_per_tablet, size_based_scanners_per_tablet));
- int num_ranges = ranges->size();
- for (int i = 0; i < num_ranges;) {
- std::vector<OlapScanRange*> scanner_ranges;
- scanner_ranges.push_back((*ranges)[i].get());
- ++i;
- for (int j = 1; i < num_ranges && j < ranges_per_scanner &&
- (*ranges)[i]->end_include == (*ranges)[i - 1]->end_include;
- ++j, ++i) {
- scanner_ranges.push_back((*ranges)[i].get());
- }
- VOlapScanner* scanner = new VOlapScanner(state, this, _olap_scan_node.is_preaggregation,
- _need_agg_finalize, *scan_range);
- // add scanner to pool before doing prepare.
- // so that scanner can be automatically deconstructed if prepare failed.
- _scanner_pool.add(scanner);
- RETURN_IF_ERROR(scanner->prepare(*scan_range, scanner_ranges, _olap_filter,
- _bloom_filters_push_down, _push_down_functions));
-
- _volap_scanners.push_back(scanner);
- disk_set.insert(scanner->scan_disk());
- }
- }
- COUNTER_SET(_num_disks_accessed_counter, static_cast<int64_t>(disk_set.size()));
- COUNTER_SET(_num_scanners, static_cast<int64_t>(_volap_scanners.size()));
-
- // init progress
- std::stringstream ss;
- ss << "ScanThread complete (node=" << id() << "):";
- _progress = ProgressUpdater(ss.str(), _volap_scanners.size(), 1);
-
- _transfer_thread.reset(new std::thread(
- [this, state, parent_span = opentelemetry::trace::Tracer::GetCurrentSpan()] {
- opentelemetry::trace::Scope scope {parent_span};
- transfer_thread(state);
- }));
-
- return Status::OK();
-}
-
-Status VOlapScanNode::close(RuntimeState* state) {
- if (is_closed()) {
- return Status::OK();
- }
- START_AND_SCOPE_SPAN(state->get_tracer(), span, "VOlapScanNode::close");
- // change done status
- {
- std::unique_lock<std::mutex> l(_blocks_lock);
- _transfer_done = true;
- }
- // notify all scanner thread
- _block_consumed_cv.notify_all();
- _block_added_cv.notify_all();
- _scan_block_added_cv.notify_all();
-
- // join transfer thread
- if (_transfer_thread) {
- _transfer_thread->join();
- }
-
- // clear some block in queue
- // TODO: The presence of transfer_thread here may cause Block's memory alloc and be released not in a thread,
- // which may lead to potential performance problems. we should rethink whether to delete the transfer thread
- std::for_each(_materialized_blocks.begin(), _materialized_blocks.end(),
- std::default_delete<Block>());
- _materialized_row_batches_bytes = 0;
- std::for_each(_scan_blocks.begin(), _scan_blocks.end(), std::default_delete<Block>());
- _scan_row_batches_bytes = 0;
- std::for_each(_free_blocks.begin(), _free_blocks.end(), std::default_delete<Block>());
-
- // OlapScanNode terminate by exception
- // so that initiative close the Scanner
- for (auto scanner : _volap_scanners) {
- scanner->close(state);
- }
-
- for (auto& filter_ctx : _runtime_filter_ctxs) {
- filter_ctx.runtimefilter->consumer_close();
- }
-
- for (auto& ctx : _stale_vexpr_ctxs) {
- (*ctx)->close(state);
- }
-
- VLOG_CRITICAL << "VOlapScanNode::close()";
- return ScanNode::close(state);
-}
-
-Status VOlapScanNode::get_next(RuntimeState* state, Block* block, bool* eos) {
- INIT_AND_SCOPE_GET_NEXT_SPAN(state->get_tracer(), _get_next_span, "VOlapScanNode::get_next");
- SCOPED_TIMER(_runtime_profile->total_time_counter());
- SCOPED_CONSUME_MEM_TRACKER(mem_tracker());
-
- // check if Canceled.
- if (state->is_cancelled()) {
- std::unique_lock<std::mutex> l(_blocks_lock);
- _transfer_done = true;
- std::lock_guard<SpinLock> guard(_status_mutex);
- if (LIKELY(_status.ok())) {
- _status = Status::Cancelled("Cancelled");
- }
- return _status;
- }
-
- // check if started.
- if (!_start) {
- Status status = start_scan(state);
-
- if (!status.ok()) {
- LOG(ERROR) << "StartScan Failed cause " << status.get_error_msg();
- *eos = true;
- return status;
- }
-
- _start = true;
- }
-
- // some conjuncts will be disposed in start_scan function, so
- // we should check _eos after call start_scan
- if (_eos) {
- *eos = true;
- return Status::OK();
- }
-
- // wait for block from queue
- Block* materialized_block = nullptr;
- {
- std::unique_lock<std::mutex> l(_blocks_lock);
- SCOPED_TIMER(_olap_wait_batch_queue_timer);
- while (_materialized_blocks.empty() && !_transfer_done) {
- if (state->is_cancelled()) {
- _transfer_done = true;
- }
-
- // use wait_for, not wait, in case to capture the state->is_cancelled()
- _block_added_cv.wait_for(l, std::chrono::seconds(1));
- }
-
- if (!_materialized_blocks.empty()) {
- materialized_block = _materialized_blocks.back();
- DCHECK(materialized_block != nullptr);
- _materialized_blocks.pop_back();
- _materialized_row_batches_bytes -= materialized_block->allocated_bytes();
- }
- }
-
- // return block
- if (nullptr != materialized_block) {
- // notify scanner
- _block_consumed_cv.notify_one();
- // get scanner's block memory
- block->swap(*materialized_block);
- VLOG_ROW << "VOlapScanNode output rows: " << block->rows();
- reached_limit(block, eos);
-
- // reach scan node limit
- if (*eos) {
- {
- std::unique_lock<std::mutex> l(_blocks_lock);
- _transfer_done = true;
- }
-
- _block_consumed_cv.notify_all();
- *eos = true;
- } else {
- *eos = false;
- }
-
- {
- // ReThink whether the SpinLock Better
- std::lock_guard<std::mutex> l(_free_blocks_lock);
- _free_blocks.emplace_back(materialized_block);
- }
- return Status::OK();
- }
-
- // all scanner done, change *eos to true
- *eos = true;
- std::lock_guard<SpinLock> guard(_status_mutex);
- return _status;
-}
-
-Block* VOlapScanNode::_alloc_block(bool& get_free_block) {
- {
- std::lock_guard<std::mutex> l(_free_blocks_lock);
- if (!_free_blocks.empty()) {
- auto block = _free_blocks.back();
- _free_blocks.pop_back();
- return block;
- }
- }
-
- get_free_block = false;
-
- auto block = new Block(_tuple_desc->slots(), _block_size);
- _buffered_bytes += block->allocated_bytes();
- return block;
-}
-
-int VOlapScanNode::_start_scanner_thread_task(RuntimeState* state, int block_per_scanner) {
- std::list<VOlapScanner*> olap_scanners;
- int assigned_thread_num = _running_thread;
- size_t max_thread = config::doris_scanner_queue_size;
- if (config::doris_scanner_row_num > state->batch_size()) {
- max_thread /= config::doris_scanner_row_num / state->batch_size();
- if (max_thread <= 0) max_thread = 1;
- }
- // copy to local
- {
- // How many thread can apply to this query
- size_t thread_slot_num = 0;
- {
- if (_scan_row_batches_bytes < _max_scanner_queue_size_bytes / 2) {
- std::lock_guard<std::mutex> l(_free_blocks_lock);
- thread_slot_num = _free_blocks.size() / block_per_scanner;
- thread_slot_num += (_free_blocks.size() % block_per_scanner != 0);
- thread_slot_num = std::min(thread_slot_num, max_thread - assigned_thread_num);
- if (thread_slot_num <= 0) {
- thread_slot_num = 1;
- }
- } else {
- std::lock_guard<std::mutex> l(_scan_blocks_lock);
- if (_scan_blocks.empty()) {
- // Just for notify if _scan_blocks is empty and no running thread
- if (assigned_thread_num == 0) {
- thread_slot_num = 1;
- // NOTE: if olap_scanners_ is empty, scanner_done_ should be true
- }
- }
- }
- }
-
- {
- std::lock_guard<std::mutex> l(_volap_scanners_lock);
- thread_slot_num = std::min(thread_slot_num, _volap_scanners.size());
- for (int i = 0; i < thread_slot_num && !_volap_scanners.empty();) {
- auto scanner = _volap_scanners.front();
- _volap_scanners.pop_front();
-
- if (scanner->need_to_close()) {
- scanner->close(state);
- } else {
- olap_scanners.push_back(scanner);
- _running_thread++;
- assigned_thread_num++;
- i++;
- }
- }
- }
- }
-
- // post volap scanners to thread-pool
- ThreadPoolToken* thread_token = state->get_query_fragments_ctx()->get_token();
- auto iter = olap_scanners.begin();
- if (thread_token != nullptr) {
- while (iter != olap_scanners.end()) {
- auto s = thread_token->submit_func(
- [this, scanner = *iter] { this->scanner_thread(scanner); });
- if (s.ok()) {
- (*iter)->start_wait_worker_timer();
- COUNTER_UPDATE(_scanner_sched_counter, 1);
- olap_scanners.erase(iter++);
- } else {
- LOG(FATAL) << "Failed to assign scanner task to thread pool! " << s.get_error_msg();
- }
- ++_total_assign_num;
- }
- } else {
- PriorityThreadPool* thread_pool = state->exec_env()->scan_thread_pool();
- PriorityThreadPool* remote_thread_pool = state->exec_env()->remote_scan_thread_pool();
- while (iter != olap_scanners.end()) {
- PriorityThreadPool::Task task;
- task.work_function = [this, scanner = *iter] { this->scanner_thread(scanner); };
- task.priority = _nice;
- task.queue_id = state->exec_env()->store_path_to_index((*iter)->scan_disk());
- (*iter)->start_wait_worker_timer();
-
- TabletStorageType type = (*iter)->get_storage_type();
- bool ret = false;
- COUNTER_UPDATE(_scanner_sched_counter, 1);
- if (type == TabletStorageType::STORAGE_TYPE_LOCAL) {
- ret = thread_pool->offer(task);
- } else {
- ret = remote_thread_pool->offer(task);
- }
-
- if (ret) {
- olap_scanners.erase(iter++);
- } else {
- LOG(FATAL) << "Failed to assign scanner task to thread pool!";
- }
- ++_total_assign_num;
- }
- }
-
- return assigned_thread_num;
-}
-
-// PlanFragmentExecutor will call this method to set scan range
-// Doris scan range is defined in thrift file like this
-// struct TPaloScanRange {
-// 1: required list<Types.TNetworkAddress> hosts
-// 2: required string schema_hash
-// 3: required string version
-// 5: required Types.TTabletId tablet_id
-// 6: required string db_name
-// 7: optional list<TKeyRange> partition_column_ranges
-// 8: optional string index_name
-// 9: optional string table_name
-//}
-// every doris_scan_range is related with one tablet so that one olap scan node contains multiple tablet
-Status VOlapScanNode::set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) {
- for (auto& scan_range : scan_ranges) {
- DCHECK(scan_range.scan_range.__isset.palo_scan_range);
- _scan_ranges.emplace_back(new TPaloScanRange(scan_range.scan_range.palo_scan_range));
- COUNTER_UPDATE(_tablet_counter, 1);
- }
-
- return Status::OK();
-}
-
-Status VOlapScanNode::get_hints(TabletSharedPtr table, const TPaloScanRange& scan_range,
- int block_row_count, bool is_begin_include, bool is_end_include,
- const std::vector<std::unique_ptr<OlapScanRange>>& scan_key_range,
- std::vector<std::unique_ptr<OlapScanRange>>* sub_scan_range,
- RuntimeProfile* profile) {
- RuntimeProfile::Counter* show_hints_timer = profile->get_counter("ShowHintsTime_V1");
- std::vector<std::vector<OlapTuple>> ranges;
- bool have_valid_range = false;
- for (auto& key_range : scan_key_range) {
- if (key_range->begin_scan_range.size() == 1 &&
- key_range->begin_scan_range.get_value(0) == NEGATIVE_INFINITY) {
- continue;
- }
- SCOPED_TIMER(show_hints_timer);
-
- Status res = Status::OK();
- std::vector<OlapTuple> range;
- res = table->split_range(key_range->begin_scan_range, key_range->end_scan_range,
- block_row_count, &range);
- if (!res.ok()) {
- return Status::InternalError("fail to show hints");
- }
- ranges.emplace_back(std::move(range));
- have_valid_range = true;
- }
-
- if (!have_valid_range) {
- std::vector<OlapTuple> range;
- auto res = table->split_range({}, {}, block_row_count, &range);
- if (!res.ok()) {
- return Status::InternalError("fail to show hints");
- }
- ranges.emplace_back(std::move(range));
- }
-
- for (int i = 0; i < ranges.size(); ++i) {
- for (int j = 0; j < ranges[i].size(); j += 2) {
- std::unique_ptr<OlapScanRange> range(new OlapScanRange);
- range->begin_scan_range.reset();
- range->begin_scan_range = ranges[i][j];
- range->end_scan_range.reset();
- range->end_scan_range = ranges[i][j + 1];
-
- if (0 == j) {
- range->begin_include = is_begin_include;
- } else {
- range->begin_include = true;
- }
-
- if (j + 2 == ranges[i].size()) {
- range->end_include = is_end_include;
- } else {
- range->end_include = false;
- }
-
- sub_scan_range->emplace_back(std::move(range));
- }
- }
-
- return Status::OK();
-}
-
-template <bool IsNotIn>
-bool VOlapScanNode::_should_push_down_in_predicate(VInPredicate* pred, VExprContext* expr_ctx) {
- if (pred->is_not_in() != IsNotIn) {
- return false;
- }
- InState* state = reinterpret_cast<InState*>(
- expr_ctx->fn_context(pred->fn_context_index())
- ->get_function_state(FunctionContext::FRAGMENT_LOCAL));
- HybridSetBase* set = state->hybrid_set.get();
-
- // if there are too many elements in InPredicate, exceed the limit,
- // we will not push any condition of this column to storage engine.
- // because too many conditions pushed down to storage engine may even
- // slow down the query process.
- // ATTN: This is just an experience value. You may need to try
- // different thresholds to improve performance.
- if (set->size() > _max_pushdown_conditions_per_column) {
- VLOG_NOTICE << "Predicate value num " << set->size() << " exceed limit "
- << _max_pushdown_conditions_per_column;
- return false;
- }
- return true;
-}
-
-bool VOlapScanNode::_should_push_down_function_filter(VectorizedFnCall* fn_call,
- VExprContext* expr_ctx,
- StringVal* constant_str,
- doris_udf::FunctionContext** fn_ctx) {
- // Now only `like` function filters is supported to push down
- if (fn_call->fn().name.function_name != "like") {
- return false;
- }
-
- const auto& children = fn_call->children();
- doris_udf::FunctionContext* func_cxt = expr_ctx->fn_context(fn_call->fn_context_index());
- DCHECK(func_cxt != nullptr);
- DCHECK(children.size() == 2);
- for (size_t i = 0; i < children.size(); i++) {
- if (VExpr::expr_without_cast(children[i])->node_type() != TExprNodeType::SLOT_REF) {
- // not a slot ref(column)
- continue;
- }
- if (!children[1 - i]->is_constant()) {
- // only handle constant value
- return false;
- } else {
- DCHECK(children[1 - i]->type().is_string_type());
- if (const ColumnConst* const_column = check_and_get_column<ColumnConst>(
- children[1 - i]->get_const_col(expr_ctx)->column_ptr)) {
- *constant_str = const_column->get_data_at(0).to_string_val();
- } else {
- return false;
- }
- }
- }
- *fn_ctx = func_cxt;
- return true;
-}
-
-bool VOlapScanNode::_should_push_down_binary_predicate(
- VectorizedFnCall* fn_call, VExprContext* expr_ctx, StringRef* constant_val,
- int* slot_ref_child, const std::function<bool(const std::string&)>& fn_checker) {
- if (!fn_checker(fn_call->fn().name.function_name)) {
- return false;
- }
-
- const auto& children = fn_call->children();
- DCHECK(children.size() == 2);
- for (size_t i = 0; i < children.size(); i++) {
- if (VExpr::expr_without_cast(children[i])->node_type() != TExprNodeType::SLOT_REF) {
- // not a slot ref(column)
- continue;
- }
- if (!children[1 - i]->is_constant()) {
- // only handle constant value
- return false;
- } else {
- if (const ColumnConst* const_column = check_and_get_column<ColumnConst>(
- children[1 - i]->get_const_col(expr_ctx)->column_ptr)) {
- *slot_ref_child = i;
- *constant_val = const_column->get_data_at(0);
- } else {
- return false;
- }
- }
- }
- return true;
-}
-
-bool VOlapScanNode::_is_predicate_acting_on_slot(
- VExpr* expr,
- const std::function<bool(const std::vector<VExpr*>&, const VSlotRef**, VExpr**)>& checker,
- SlotDescriptor** slot_desc, ColumnValueRangeType** range) {
- const VSlotRef* slot_ref = nullptr;
- VExpr* child_contains_slot = nullptr;
- if (!checker(expr->children(), &slot_ref, &child_contains_slot)) {
- // not a slot ref(column)
- return false;
- }
-
- auto entry = _id_to_slot_column_value_range.find(slot_ref->slot_id());
- if (_id_to_slot_column_value_range.end() == entry) {
- return false;
- }
- *slot_desc = entry->second.first;
- DCHECK(child_contains_slot != nullptr);
- if (child_contains_slot->type().type != (*slot_desc)->type().type) {
- if (!ignore_cast(*slot_desc, child_contains_slot)) {
- // the type of predicate not match the slot's type
- return false;
- }
- }
- *range = &(entry->second.second);
- return true;
-}
-
-template <PrimitiveType T>
-Status VOlapScanNode::_normalize_in_and_eq_predicate(VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot,
- ColumnValueRange<T>& range, bool* push_down) {
- auto temp_range = ColumnValueRange<T>::create_empty_column_value_range(slot->type().precision,
- slot->type().scale);
- bool effect = false;
- // 1. Normalize in conjuncts like 'where col in (v1, v2, v3)'
- if (TExprNodeType::IN_PRED == expr->node_type()) {
- VInPredicate* pred = static_cast<VInPredicate*>(expr);
- if (!_should_push_down_in_predicate<false>(pred, expr_ctx)) {
- return Status::OK();
- }
-
- // begin to push InPredicate value into ColumnValueRange
- InState* state = reinterpret_cast<InState*>(
- expr_ctx->fn_context(pred->fn_context_index())
- ->get_function_state(FunctionContext::FRAGMENT_LOCAL));
- HybridSetBase::IteratorBase* iter = state->hybrid_set->begin();
- auto fn_name = std::string("");
- while (iter->has_next()) {
- // column in (nullptr) is always false so continue to
- // dispose next item
- if (nullptr == iter->get_value()) {
- iter->next();
- continue;
- }
- auto value = const_cast<void*>(iter->get_value());
- RETURN_IF_ERROR(change_value_range<true>(
- temp_range, value, ColumnValueRange<T>::add_fixed_value_range, fn_name));
- iter->next();
- }
-
- range.intersection(temp_range);
- effect = true;
- } else if (TExprNodeType::BINARY_PRED == expr->node_type()) {
- DCHECK(expr->children().size() == 2);
- auto eq_checker = [](const std::string& fn_name) { return fn_name == "eq"; };
-
- StringRef value;
- int slot_ref_child = -1;
- if (_should_push_down_binary_predicate(reinterpret_cast<VectorizedFnCall*>(expr), expr_ctx,
- &value, &slot_ref_child, eq_checker)) {
- DCHECK(slot_ref_child >= 0);
- // where A = nullptr should return empty result set
- auto fn_name = std::string("");
- if (value.data != nullptr) {
- if constexpr (T == TYPE_CHAR || T == TYPE_VARCHAR || T == TYPE_STRING ||
- T == TYPE_HLL) {
- auto val = StringValue(value.data, value.size);
- RETURN_IF_ERROR(change_value_range<true>(
- temp_range, reinterpret_cast<void*>(&val),
- ColumnValueRange<T>::add_fixed_value_range, fn_name));
- } else {
- RETURN_IF_ERROR(change_value_range<true>(
- temp_range, reinterpret_cast<void*>(const_cast<char*>(value.data)),
- ColumnValueRange<T>::add_fixed_value_range, fn_name));
- }
- range.intersection(temp_range);
- effect = true;
- }
- }
- }
-
- // exceed limit, no conditions will be pushed down to storage engine.
- if (range.get_fixed_value_size() > _max_pushdown_conditions_per_column) {
- range.set_whole_value_range();
- } else {
- *push_down = effect;
- }
- return Status::OK();
-}
-
-template <PrimitiveType T>
-Status VOlapScanNode::_normalize_not_in_and_not_eq_predicate(VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot,
- ColumnValueRange<T>& range,
- bool* push_down) {
- bool is_fixed_range = range.is_fixed_value_range();
- auto not_in_range = ColumnValueRange<T>::create_empty_column_value_range(range.column_name());
- bool effect = false;
- // 1. Normalize in conjuncts like 'where col in (v1, v2, v3)'
- if (TExprNodeType::IN_PRED == expr->node_type()) {
- VInPredicate* pred = static_cast<VInPredicate*>(expr);
- if (!_should_push_down_in_predicate<true>(pred, expr_ctx)) {
- return Status::OK();
- }
-
- // begin to push InPredicate value into ColumnValueRange
- InState* state = reinterpret_cast<InState*>(
- expr_ctx->fn_context(pred->fn_context_index())
- ->get_function_state(FunctionContext::FRAGMENT_LOCAL));
- HybridSetBase::IteratorBase* iter = state->hybrid_set->begin();
- auto fn_name = std::string("");
- while (iter->has_next()) {
- // column not in (nullptr) is always true
- if (nullptr == iter->get_value()) {
- continue;
- }
- auto value = const_cast<void*>(iter->get_value());
- if (is_fixed_range) {
- RETURN_IF_ERROR(change_value_range<true>(
- range, value, ColumnValueRange<T>::remove_fixed_value_range, fn_name));
- } else {
- RETURN_IF_ERROR(change_value_range<true>(
- not_in_range, value, ColumnValueRange<T>::add_fixed_value_range, fn_name));
- }
- iter->next();
- }
- effect = true;
- } else if (TExprNodeType::BINARY_PRED == expr->node_type()) {
- DCHECK(expr->children().size() == 2);
-
- auto ne_checker = [](const std::string& fn_name) { return fn_name == "ne"; };
- StringRef value;
- int slot_ref_child = -1;
- if (_should_push_down_binary_predicate(reinterpret_cast<VectorizedFnCall*>(expr), expr_ctx,
- &value, &slot_ref_child, ne_checker)) {
- DCHECK(slot_ref_child >= 0);
- // where A = nullptr should return empty result set
- if (value.data != nullptr) {
- auto fn_name = std::string("");
- if constexpr (T == TYPE_CHAR || T == TYPE_VARCHAR || T == TYPE_STRING ||
- T == TYPE_HLL) {
- auto val = StringValue(value.data, value.size);
- if (is_fixed_range) {
- RETURN_IF_ERROR(change_value_range<true>(
- range, reinterpret_cast<void*>(&val),
- ColumnValueRange<T>::remove_fixed_value_range, fn_name));
- } else {
- RETURN_IF_ERROR(change_value_range<true>(
- not_in_range, reinterpret_cast<void*>(&val),
- ColumnValueRange<T>::add_fixed_value_range, fn_name));
- }
- } else {
- if (is_fixed_range) {
- RETURN_IF_ERROR(change_value_range<true>(
- range, reinterpret_cast<void*>(const_cast<char*>(value.data)),
- ColumnValueRange<T>::remove_fixed_value_range, fn_name));
- } else {
- RETURN_IF_ERROR(change_value_range<true>(
- not_in_range,
- reinterpret_cast<void*>(const_cast<char*>(value.data)),
- ColumnValueRange<T>::add_fixed_value_range, fn_name));
- }
- }
- effect = true;
- }
- }
- }
-
- if (is_fixed_range ||
- not_in_range.get_fixed_value_size() <= _max_pushdown_conditions_per_column) {
- if (!is_fixed_range) {
- // push down not in condition to storage engine
- not_in_range.to_in_condition(_olap_filter, false);
- }
- *push_down = effect;
- }
- return Status::OK();
-}
-
-template <PrimitiveType T>
-Status VOlapScanNode::_normalize_is_null_predicate(VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, ColumnValueRange<T>& range,
- bool* push_down) {
- if (TExprNodeType::FUNCTION_CALL == expr->node_type()) {
- if (reinterpret_cast<VectorizedFnCall*>(expr)->fn().name.function_name == "is_null_pred") {
- auto temp_range = ColumnValueRange<T>::create_empty_column_value_range(
- slot->type().precision, slot->type().scale);
- temp_range.set_contain_null(true);
- range.intersection(temp_range);
- *push_down = true;
- } else if (reinterpret_cast<VectorizedFnCall*>(expr)->fn().name.function_name ==
- "is_not_null_pred") {
- auto temp_range = ColumnValueRange<T>::create_empty_column_value_range(
- slot->type().precision, slot->type().scale);
- temp_range.set_contain_null(false);
- range.intersection(temp_range);
- *push_down = true;
- }
- }
- return Status::OK();
-}
-
-template <PrimitiveType T>
-Status VOlapScanNode::_normalize_noneq_binary_predicate(VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot,
- ColumnValueRange<T>& range,
- bool* push_down) {
- if (TExprNodeType::BINARY_PRED == expr->node_type()) {
- DCHECK(expr->children().size() == 2);
-
- auto noneq_checker = [](const std::string& fn_name) {
- return fn_name != "ne" && fn_name != "eq";
- };
- StringRef value;
- int slot_ref_child = -1;
- if (_should_push_down_binary_predicate(reinterpret_cast<VectorizedFnCall*>(expr), expr_ctx,
- &value, &slot_ref_child, noneq_checker)) {
- DCHECK(slot_ref_child >= 0);
- const std::string& fn_name =
- reinterpret_cast<VectorizedFnCall*>(expr)->fn().name.function_name;
-
- // where A = nullptr should return empty result set
- if (value.data != nullptr) {
- *push_down = true;
- if constexpr (T == TYPE_CHAR || T == TYPE_VARCHAR || T == TYPE_STRING ||
- T == TYPE_HLL) {
- auto val = StringValue(value.data, value.size);
- RETURN_IF_ERROR(change_value_range<false>(range, reinterpret_cast<void*>(&val),
- ColumnValueRange<T>::add_value_range,
- fn_name, slot_ref_child));
- } else {
- RETURN_IF_ERROR(change_value_range<false>(
- range, reinterpret_cast<void*>(const_cast<char*>(value.data)),
- ColumnValueRange<T>::add_value_range, fn_name, slot_ref_child));
- }
- }
- }
- }
- return Status::OK();
-}
-
-Status VOlapScanNode::_normalize_bloom_filter(VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, bool* push_down) {
- if (TExprNodeType::BLOOM_PRED == expr->node_type()) {
- DCHECK(expr->children().size() == 1);
- _bloom_filters_push_down.emplace_back(slot->col_name(), expr->get_bloom_filter_func());
- *push_down = true;
- }
- return Status::OK();
-}
-
-Status VOlapScanNode::_normalize_function_filters(VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, bool* push_down) {
- bool opposite = false;
- VExpr* fn_expr = expr;
- if (TExprNodeType::COMPOUND_PRED == expr->node_type() &&
- expr->fn().name.function_name == "not") {
- fn_expr = fn_expr->children()[0];
- opposite = true;
- }
-
- if (TExprNodeType::FUNCTION_CALL == fn_expr->node_type()) {
- doris_udf::FunctionContext* fn_ctx = nullptr;
- StringVal val;
- if (_should_push_down_function_filter(reinterpret_cast<VectorizedFnCall*>(fn_expr),
- expr_ctx, &val, &fn_ctx)) {
- std::string col = slot->col_name();
- _push_down_functions.emplace_back(opposite, col, fn_ctx, val);
- *push_down = true;
- }
- }
- return Status::OK();
-}
-
-void VOlapScanNode::eval_const_conjuncts(VExpr* vexpr, VExprContext* expr_ctx, bool* push_down) {
- char* constant_val = nullptr;
- if (vexpr->is_constant()) {
- if (const ColumnConst* const_column =
- check_and_get_column<ColumnConst>(vexpr->get_const_col(expr_ctx)->column_ptr)) {
- constant_val = const_cast<char*>(const_column->get_data_at(0).data);
- if (constant_val == nullptr || *reinterpret_cast<bool*>(constant_val) == false) {
- *push_down = true;
- _eos = true;
- }
- } else if (const ColumnVector<UInt8>* bool_column =
- check_and_get_column<ColumnVector<UInt8>>(
- vexpr->get_const_col(expr_ctx)->column_ptr)) {
- // TODO: If `vexpr->is_constant()` is true, a const column is expected here.
- // But now we still don't cover all predicates for const expression.
- // For example, for query `SELECT col FROM tbl WHERE 'PROMOTION' LIKE 'AAA%'`,
- // predicate `like` will return a ColumnVector<UInt8> which contains a single value.
- LOG(WARNING) << "Expr[" << vexpr->debug_string()
- << "] should return a const column but actually is "
- << vexpr->get_const_col(expr_ctx)->column_ptr->get_name();
- DCHECK_EQ(bool_column->size(), 1);
- if (bool_column->size() == 1) {
- constant_val = const_cast<char*>(bool_column->get_data_at(0).data);
- if (constant_val == nullptr || *reinterpret_cast<bool*>(constant_val) == false) {
- *push_down = true;
- _eos = true;
- }
- } else {
- LOG(WARNING) << "Constant predicate in scan node should return a bool column with "
- "`size == 1` but actually is "
- << bool_column->size();
- }
- } else {
- LOG(WARNING) << "Expr[" << vexpr->debug_string()
- << "] should return a boolean column but actually is "
- << vexpr->get_const_col(expr_ctx)->column_ptr->get_name();
- }
- }
-}
-
-VExpr* VOlapScanNode::_normalize_predicate(RuntimeState* state, VExpr* conjunct_expr_root) {
- static constexpr auto is_leaf = [](VExpr* expr) { return !expr->is_and_expr(); };
- auto in_predicate_checker = [](const std::vector<VExpr*>& children, const VSlotRef** slot,
- VExpr** child_contains_slot) {
- if (children.empty() ||
- VExpr::expr_without_cast(children[0])->node_type() != TExprNodeType::SLOT_REF) {
- // not a slot ref(column)
- return false;
- }
- *slot = reinterpret_cast<const VSlotRef*>(VExpr::expr_without_cast(children[0]));
- *child_contains_slot = children[0];
- return true;
- };
- auto eq_predicate_checker = [](const std::vector<VExpr*>& children, const VSlotRef** slot,
- VExpr** child_contains_slot) {
- for (const VExpr* child : children) {
- if (VExpr::expr_without_cast(child)->node_type() != TExprNodeType::SLOT_REF) {
- // not a slot ref(column)
- continue;
- }
- *slot = reinterpret_cast<const VSlotRef*>(VExpr::expr_without_cast(child));
- *child_contains_slot = const_cast<VExpr*>(child);
- return true;
- }
- return false;
- };
-
- if (conjunct_expr_root != nullptr) {
- if (is_leaf(conjunct_expr_root)) {
- auto impl = conjunct_expr_root->get_impl();
- VExpr* cur_expr = impl ? const_cast<VExpr*>(impl) : conjunct_expr_root;
- SlotDescriptor* slot = nullptr;
- ColumnValueRangeType* range = nullptr;
- bool push_down = false;
- eval_const_conjuncts(cur_expr, *(_vconjunct_ctx_ptr.get()), &push_down);
- if (push_down) {
- return nullptr;
- }
- if (_is_predicate_acting_on_slot(cur_expr, in_predicate_checker, &slot, &range) ||
- _is_predicate_acting_on_slot(cur_expr, eq_predicate_checker, &slot, &range)) {
- std::visit(
- [&](auto& value_range) {
- RETURN_IF_PUSH_DOWN(_normalize_in_and_eq_predicate(
- cur_expr, *(_vconjunct_ctx_ptr.get()), slot, value_range,
- &push_down));
- RETURN_IF_PUSH_DOWN(_normalize_not_in_and_not_eq_predicate(
- cur_expr, *(_vconjunct_ctx_ptr.get()), slot, value_range,
- &push_down));
- RETURN_IF_PUSH_DOWN(_normalize_is_null_predicate(
- cur_expr, *(_vconjunct_ctx_ptr.get()), slot, value_range,
- &push_down));
- RETURN_IF_PUSH_DOWN(_normalize_noneq_binary_predicate(
- cur_expr, *(_vconjunct_ctx_ptr.get()), slot, value_range,
- &push_down));
- if (is_key_column(slot->col_name())) {
- RETURN_IF_PUSH_DOWN(_normalize_bloom_filter(
- cur_expr, *(_vconjunct_ctx_ptr.get()), slot, &push_down));
- if (state->enable_function_pushdown()) {
- RETURN_IF_PUSH_DOWN(_normalize_function_filters(
- cur_expr, *(_vconjunct_ctx_ptr.get()), slot,
- &push_down));
- }
- }
- },
- *range);
- }
- if (push_down && is_key_column(slot->col_name())) {
- return nullptr;
- } else {
- return conjunct_expr_root;
- }
- } else {
- VExpr* left_child = _normalize_predicate(state, conjunct_expr_root->children()[0]);
- VExpr* right_child = _normalize_predicate(state, conjunct_expr_root->children()[1]);
-
- if (left_child != nullptr && right_child != nullptr) {
- conjunct_expr_root->set_children({left_child, right_child});
- return conjunct_expr_root;
- } else {
- // here only close the and expr self, do not close the child
- conjunct_expr_root->set_children({});
- conjunct_expr_root->close(state, *_vconjunct_ctx_ptr,
- (*_vconjunct_ctx_ptr)->get_function_state_scope());
- }
-
- // here do not close Expr* now
- return left_child != nullptr ? left_child : right_child;
- }
- }
- return conjunct_expr_root;
-}
-
-Status VOlapScanNode::_append_rf_into_conjuncts(RuntimeState* state, std::vector<VExpr*>& vexprs) {
- if (!vexprs.empty()) {
- VExpr* last_expr = nullptr;
- if (_vconjunct_ctx_ptr) {
- last_expr = (*_vconjunct_ctx_ptr)->root();
- } else {
- DCHECK(_rf_vexpr_set.find(vexprs[0]) == _rf_vexpr_set.end());
- last_expr = vexprs[0];
- _rf_vexpr_set.insert(vexprs[0]);
- }
- for (size_t j = _vconjunct_ctx_ptr ? 0 : 1; j < vexprs.size(); j++) {
- if (_rf_vexpr_set.find(vexprs[j]) != _rf_vexpr_set.end()) {
- continue;
- }
- TFunction fn;
- TFunctionName fn_name;
- fn_name.__set_db_name("");
- fn_name.__set_function_name("and");
- fn.__set_name(fn_name);
- fn.__set_binary_type(TFunctionBinaryType::BUILTIN);
- std::vector<TTypeDesc> arg_types;
- arg_types.push_back(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
- arg_types.push_back(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
- fn.__set_arg_types(arg_types);
- fn.__set_ret_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
- fn.__set_has_var_args(false);
-
- TExprNode texpr_node;
- texpr_node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
- texpr_node.__set_node_type(TExprNodeType::COMPOUND_PRED);
- texpr_node.__set_opcode(TExprOpcode::COMPOUND_AND);
- texpr_node.__set_fn(fn);
- texpr_node.__set_is_nullable(last_expr->is_nullable() || vexprs[j]->is_nullable());
- VExpr* new_node = _pool->add(new VcompoundPred(texpr_node));
- new_node->add_child(last_expr);
- DCHECK((vexprs[j])->get_impl() != nullptr);
- new_node->add_child(vexprs[j]);
- last_expr = new_node;
- _rf_vexpr_set.insert(vexprs[j]);
- }
- auto new_vconjunct_ctx_ptr = _pool->add(new VExprContext(last_expr));
- if (_vconjunct_ctx_ptr) {
- (*_vconjunct_ctx_ptr)->clone_fn_contexts(new_vconjunct_ctx_ptr);
- }
- RETURN_IF_ERROR(new_vconjunct_ctx_ptr->prepare(state, _row_descriptor));
- RETURN_IF_ERROR(new_vconjunct_ctx_ptr->open(state));
- if (_vconjunct_ctx_ptr) {
- (*(_vconjunct_ctx_ptr.get()))->mark_as_stale();
- _stale_vexpr_ctxs.push_back(std::move(_vconjunct_ctx_ptr));
- }
- _vconjunct_ctx_ptr.reset(new doris::vectorized::VExprContext*);
- *(_vconjunct_ctx_ptr.get()) = new_vconjunct_ctx_ptr;
- }
- return Status::OK();
-}
-
-std::string VOlapScanNode::get_name() {
- return fmt::format("VOlapScanNode({0})", _olap_scan_node.table_name);
-}
-
-} // namespace doris::vectorized
diff --git a/be/src/vec/exec/volap_scan_node.h b/be/src/vec/exec/volap_scan_node.h
deleted file mode 100644
index 51f4ed97c0..0000000000
--- a/be/src/vec/exec/volap_scan_node.h
+++ /dev/null
@@ -1,346 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include "exec/olap_common.h"
-#include "exec/scan_node.h"
-#include "exprs/bloomfilter_predicate.h"
-#include "exprs/function_filter.h"
-#include "exprs/in_predicate.h"
-#include "exprs/runtime_filter.h"
-#include "gen_cpp/PlanNodes_types.h"
-#include "olap/tablet.h"
-#include "util/progress_updater.h"
-#include "vec/exprs/vectorized_fn_call.h"
-#include "vec/exprs/vin_predicate.h"
-#include "vec/exprs/vslot_ref.h"
-
-namespace doris {
-class ObjectPool;
-class TPlanNode;
-class DescriptorTbl;
-class RowBatch;
-namespace vectorized {
-
-class VOlapScanner;
-
-class VOlapScanNode final : public ScanNode {
-public:
- VOlapScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs);
- friend class VOlapScanner;
-
- Status init(const TPlanNode& tnode, RuntimeState* state = nullptr) override;
- Status prepare(RuntimeState* state) override;
- Status open(RuntimeState* state) override;
-
- Status get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) override {
- return Status::NotSupported("Not Implemented VOlapScanNode Node::get_next scalar");
- }
- Status get_next(RuntimeState* state, Block* block, bool* eos) override;
- Status close(RuntimeState* state) override;
-
- Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) override;
-
- Status get_hints(TabletSharedPtr table, const TPaloScanRange& scan_range, int block_row_count,
- bool is_begin_include, bool is_end_include,
- const std::vector<std::unique_ptr<OlapScanRange>>& scan_key_range,
- std::vector<std::unique_ptr<OlapScanRange>>* sub_scan_range,
- RuntimeProfile* profile);
-
- std::string get_name() override;
-
-private:
- // In order to ensure the accuracy of the query result
- // only key column conjuncts will be remove as idle conjunct
- bool is_key_column(const std::string& key_name);
-
- Status start_scan(RuntimeState* state);
- Status normalize_conjuncts();
- Status build_key_ranges_and_filters();
-
- template <bool IsFixed, PrimitiveType PrimitiveType, typename ChangeFixedValueRangeFunc>
- static Status change_value_range(ColumnValueRange<PrimitiveType>& range, void* value,
- const ChangeFixedValueRangeFunc& func,
- const std::string& fn_name, int slot_ref_child = -1);
-
- void transfer_thread(RuntimeState* state);
- void scanner_thread(VOlapScanner* scanner);
- Status start_scan_thread(RuntimeState* state);
-
- Status _add_blocks(std::vector<Block*>& block);
- int _start_scanner_thread_task(RuntimeState* state, int block_per_scanner);
- Block* _alloc_block(bool& get_free_block);
-
- void _init_counter(RuntimeState* state);
- // OLAP_SCAN_NODE profile layering: OLAP_SCAN_NODE, OlapScanner, and SegmentIterator
- // according to the calling relationship
- void init_scan_profile();
- const std::vector<TRuntimeFilterDesc>& runtime_filter_descs() const {
- return _runtime_filter_descs;
- }
-
- template <PrimitiveType T>
- Status _normalize_in_and_eq_predicate(vectorized::VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, ColumnValueRange<T>& range,
- bool* push_down);
- template <PrimitiveType T>
- Status _normalize_not_in_and_not_eq_predicate(vectorized::VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, ColumnValueRange<T>& range,
- bool* push_down);
-
- template <PrimitiveType T>
- Status _normalize_noneq_binary_predicate(vectorized::VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, ColumnValueRange<T>& range,
- bool* push_down);
-
- template <PrimitiveType T>
- Status _normalize_is_null_predicate(vectorized::VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, ColumnValueRange<T>& range,
- bool* push_down);
-
- Status _normalize_bloom_filter(vectorized::VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, bool* push_down);
-
- void eval_const_conjuncts(VExpr* vexpr, VExprContext* expr_ctx, bool* push_down);
-
- vectorized::VExpr* _normalize_predicate(RuntimeState* state,
- vectorized::VExpr* conjunct_expr_root);
-
- Status _normalize_function_filters(vectorized::VExpr* expr, VExprContext* expr_ctx,
- SlotDescriptor* slot, bool* push_down);
-
- template <bool IsNotIn>
- bool _should_push_down_in_predicate(VInPredicate* in_pred, VExprContext* expr_ctx);
-
- bool _is_predicate_acting_on_slot(VExpr* expr,
- const std::function<bool(const std::vector<VExpr*>&,
- const VSlotRef**, VExpr**)>& checker,
- SlotDescriptor** slot_desc, ColumnValueRangeType** range);
-
- bool _should_push_down_binary_predicate(
- VectorizedFnCall* fn_call, VExprContext* expr_ctx, StringRef* constant_val,
- int* slot_ref_child, const std::function<bool(const std::string&)>& fn_checker);
-
- bool _should_push_down_function_filter(VectorizedFnCall* fn_call, VExprContext* expr_ctx,
- StringVal* constant_str,
- doris_udf::FunctionContext** fn_ctx);
-
- Status _append_rf_into_conjuncts(RuntimeState* state, std::vector<VExpr*>& vexprs);
-
- // Tuple id resolved in prepare() to set _tuple_desc;
- TupleId _tuple_id;
- // doris scan node used to scan doris
- TOlapScanNode _olap_scan_node;
- // tuple descriptors
- const TupleDescriptor* _tuple_desc;
- // tuple index
- int _tuple_idx;
- // string slots
- std::vector<SlotDescriptor*> _string_slots;
- // collection slots
- std::vector<SlotDescriptor*> _collection_slots;
-
- bool _eos;
-
- // column -> ColumnValueRange map
- std::map<std::string, ColumnValueRangeType> _column_value_ranges;
-
- OlapScanKeys _scan_keys;
-
- std::vector<std::unique_ptr<TPaloScanRange>> _scan_ranges;
-
- std::vector<TCondition> _olap_filter;
- // push down bloom filters to storage engine.
- // 1. std::pair.first :: column name
- // 2. std::pair.second :: shared_ptr of BloomFilterFuncBase
- std::vector<std::pair<std::string, std::shared_ptr<BloomFilterFuncBase>>>
- _bloom_filters_push_down;
-
- // push down functions to storage engine
- // only support scalar functions, now just support like / not like
- std::vector<FunctionFilter> _push_down_functions;
-
- // Pool for storing allocated scanner objects. We don't want to use the
- // runtime pool to ensure that the scanner objects are deleted before this
- // object is.
- ObjectPool _scanner_pool;
-
- std::shared_ptr<std::thread> _transfer_thread;
-
- // Keeps track of total splits and the number finished.
- ProgressUpdater _progress;
-
- // to limit _materialized_row_batches_bytes < _max_scanner_queue_size_bytes / 2
- std::atomic_size_t _materialized_row_batches_bytes = 0;
-
- std::atomic_int _running_thread = 0;
- std::condition_variable _scan_thread_exit_cv;
-
- // to limit _scan_row_batches_bytes < _max_scanner_queue_size_bytes / 2
- std::atomic_size_t _scan_row_batches_bytes = 0;
-
- std::list<VOlapScanner*> _olap_scanners;
-
- int _max_materialized_row_batches;
- // to limit _materialized_row_batches_bytes and _scan_row_batches_bytes
- size_t _max_scanner_queue_size_bytes;
- bool _start;
- // Used in Scan thread to ensure thread-safe
- std::atomic_bool _scanner_done;
- std::atomic_bool _transfer_done;
-
- int _total_assign_num;
- int _nice;
-
- // protect _status, for many thread may change _status
- SpinLock _status_mutex;
- Status _status;
- RuntimeState* _runtime_state;
-
- RuntimeProfile::Counter* _scan_timer;
- RuntimeProfile::Counter* _scan_cpu_timer = nullptr;
- RuntimeProfile::Counter* _tablet_counter;
- RuntimeProfile::Counter* _rows_pushed_cond_filtered_counter = nullptr;
- RuntimeProfile::Counter* _reader_init_timer = nullptr;
- RuntimeProfile::Counter* _scanner_sched_counter = nullptr;
- TResourceInfo* _resource_info;
-
- int64_t _buffered_bytes;
- EvalConjunctsFn _eval_conjuncts_fn;
-
- // the max num of scan keys of this scan request.
- // it will set as BE's config `doris_max_scan_key_num`,
- // or be overwritten by value in TQueryOptions
- int32_t _max_scan_key_num = 1024;
- // The max number of conditions in InPredicate that can be pushed down
- // into OlapEngine.
- // If conditions in InPredicate is larger than this, all conditions in
- // InPredicate will not be pushed to the OlapEngine.
- // it will set as BE's config `max_pushdown_conditions_per_column`,
- // or be overwritten by value in TQueryOptions
- int32_t _max_pushdown_conditions_per_column = 1024;
-
- struct RuntimeFilterContext {
- RuntimeFilterContext() : apply_mark(false), runtimefilter(nullptr) {}
- bool apply_mark;
- IRuntimeFilter* runtimefilter;
- };
- std::vector<TRuntimeFilterDesc> _runtime_filter_descs;
- std::vector<RuntimeFilterContext> _runtime_filter_ctxs;
- std::vector<bool> _runtime_filter_ready_flag;
- std::shared_mutex _rf_lock;
- std::map<int, RuntimeFilterContext*> _conjunctid_to_runtime_filter_ctxs;
-
- std::unique_ptr<RuntimeProfile> _scanner_profile;
- std::unique_ptr<RuntimeProfile> _segment_profile;
-
- // Counters
- RuntimeProfile::Counter* _io_timer = nullptr;
- RuntimeProfile::Counter* _read_compressed_counter = nullptr;
- RuntimeProfile::Counter* _decompressor_timer = nullptr;
- RuntimeProfile::Counter* _read_uncompressed_counter = nullptr;
- RuntimeProfile::Counter* _raw_rows_counter = nullptr;
-
- RuntimeProfile::Counter* _rows_vec_cond_counter = nullptr;
- RuntimeProfile::Counter* _vec_cond_timer = nullptr;
- RuntimeProfile::Counter* _short_cond_timer = nullptr;
- RuntimeProfile::Counter* _output_col_timer = nullptr;
-
- RuntimeProfile::Counter* _stats_filtered_counter = nullptr;
- RuntimeProfile::Counter* _bf_filtered_counter = nullptr;
- RuntimeProfile::Counter* _del_filtered_counter = nullptr;
- RuntimeProfile::Counter* _conditions_filtered_counter = nullptr;
- RuntimeProfile::Counter* _key_range_filtered_counter = nullptr;
-
- RuntimeProfile::Counter* _block_fetch_timer = nullptr;
- RuntimeProfile::Counter* _block_load_timer = nullptr;
- RuntimeProfile::Counter* _block_load_counter = nullptr;
- // Not used any more, will be removed after non-vectorized code is removed
- RuntimeProfile::Counter* _block_seek_timer = nullptr;
- // Not used any more, will be removed after non-vectorized code is removed
- RuntimeProfile::Counter* _block_seek_counter = nullptr;
- // Add more detail seek timer and counter profile
- // Read process is split into 3 stages: init, first read, lazy read
- RuntimeProfile::Counter* _block_init_timer = nullptr;
- RuntimeProfile::Counter* _block_init_seek_timer = nullptr;
- RuntimeProfile::Counter* _block_init_seek_counter = nullptr;
- RuntimeProfile::Counter* _first_read_timer = nullptr;
- RuntimeProfile::Counter* _first_read_seek_timer = nullptr;
- RuntimeProfile::Counter* _first_read_seek_counter = nullptr;
- RuntimeProfile::Counter* _lazy_read_timer = nullptr;
- RuntimeProfile::Counter* _lazy_read_seek_timer = nullptr;
- RuntimeProfile::Counter* _lazy_read_seek_counter = nullptr;
-
- RuntimeProfile::Counter* _block_convert_timer = nullptr;
-
- RuntimeProfile::Counter* _index_load_timer = nullptr;
-
- // total pages read
- // used by segment v2
- RuntimeProfile::Counter* _total_pages_num_counter = nullptr;
- // page read from cache
- // used by segment v2
- RuntimeProfile::Counter* _cached_pages_num_counter = nullptr;
-
- // row count filtered by bitmap inverted index
- RuntimeProfile::Counter* _bitmap_index_filter_counter = nullptr;
- // time fro bitmap inverted index read and filter
- RuntimeProfile::Counter* _bitmap_index_filter_timer = nullptr;
- // number of created olap scanners
- RuntimeProfile::Counter* _num_scanners = nullptr;
-
- // number of segment filtered by column stat when creating seg iterator
- RuntimeProfile::Counter* _filtered_segment_counter = nullptr;
- // total number of segment related to this scan node
- RuntimeProfile::Counter* _total_segment_counter = nullptr;
-
- RuntimeProfile::Counter* _scanner_wait_batch_timer = nullptr;
- RuntimeProfile::Counter* _scanner_wait_worker_timer = nullptr;
-
- RuntimeProfile::Counter* _olap_wait_batch_queue_timer = nullptr;
-
- // for debugging or profiling, record any info as you want
- RuntimeProfile::Counter* _general_debug_timer[GENERAL_DEBUG_COUNT] = {};
-
- std::vector<Block*> _scan_blocks;
- std::vector<Block*> _materialized_blocks;
- std::mutex _blocks_lock;
- std::condition_variable _block_added_cv;
- std::condition_variable _block_consumed_cv;
-
- std::mutex _scan_blocks_lock;
- std::condition_variable _scan_block_added_cv;
-
- std::vector<Block*> _free_blocks;
- std::mutex _free_blocks_lock;
-
- std::list<VOlapScanner*> _volap_scanners;
- std::mutex _volap_scanners_lock;
-
- int _max_materialized_blocks;
-
- size_t _block_size = 0;
-
- phmap::flat_hash_set<VExpr*> _rf_vexpr_set;
- std::vector<std::unique_ptr<VExprContext*>> _stale_vexpr_ctxs;
- int64_t _limit_per_scanner = -1;
- phmap::flat_hash_map<int, std::pair<SlotDescriptor*, ColumnValueRangeType>>
- _id_to_slot_column_value_range;
-};
-} // namespace vectorized
-} // namespace doris
diff --git a/be/src/vec/exec/volap_scanner.cpp b/be/src/vec/exec/volap_scanner.cpp
deleted file mode 100644
index 2b313ad132..0000000000
--- a/be/src/vec/exec/volap_scanner.cpp
+++ /dev/null
@@ -1,477 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#include "vec/exec/volap_scanner.h"
-
-#include <memory>
-
-#include "olap/storage_engine.h"
-#include "runtime/runtime_state.h"
-#include "vec/core/block.h"
-#include "vec/exec/volap_scan_node.h"
-#include "vec/exprs/vexpr_context.h"
-
-namespace doris::vectorized {
-
-VOlapScanner::VOlapScanner(RuntimeState* runtime_state, VOlapScanNode* parent, bool aggregation,
- bool need_agg_finalize, const TPaloScanRange& scan_range)
- : _runtime_state(runtime_state),
- _parent(parent),
- _tuple_desc(parent->_tuple_desc),
- _id(-1),
- _is_open(false),
- _aggregation(aggregation),
- _need_agg_finalize(need_agg_finalize),
- _version(-1) {
- _tablet_schema = std::make_shared<TabletSchema>();
-}
-
-Status VOlapScanner::prepare(
- const TPaloScanRange& scan_range, const std::vector<OlapScanRange*>& key_ranges,
- const std::vector<TCondition>& filters,
- const std::vector<std::pair<string, std::shared_ptr<BloomFilterFuncBase>>>& bloom_filters,
- const std::vector<FunctionFilter>& function_filters) {
- set_tablet_reader();
- // set limit to reduce end of rowset and segment mem use
- _tablet_reader->set_batch_size(
- _parent->limit() == -1
- ? _parent->_runtime_state->batch_size()
- : std::min(static_cast<int64_t>(_parent->_runtime_state->batch_size()),
- _parent->limit()));
-
- // Get olap table
- TTabletId tablet_id = scan_range.tablet_id;
- SchemaHash schema_hash = strtoul(scan_range.schema_hash.c_str(), nullptr, 10);
- _version = strtoul(scan_range.version.c_str(), nullptr, 10);
- {
- std::string err;
- _tablet = StorageEngine::instance()->tablet_manager()->get_tablet(tablet_id, true, &err);
- if (_tablet.get() == nullptr) {
- std::stringstream ss;
- ss << "failed to get tablet. tablet_id=" << tablet_id
- << ", with schema_hash=" << schema_hash << ", reason=" << err;
- LOG(WARNING) << ss.str();
- return Status::InternalError(ss.str());
- }
- _tablet_schema->copy_from(*_tablet->tablet_schema());
- if (_parent->_olap_scan_node.__isset.columns_desc &&
- !_parent->_olap_scan_node.columns_desc.empty() &&
- _parent->_olap_scan_node.columns_desc[0].col_unique_id >= 0) {
- // Originally scanner get TabletSchema from tablet object in BE.
- // To support lightweight schema change for adding / dropping columns,
- // tabletschema is bounded to rowset and tablet's schema maybe outdated,
- // so we have to use schema from a query plan witch FE puts it in query plans.
- _tablet_schema->clear_columns();
- for (const auto& column_desc : _parent->_olap_scan_node.columns_desc) {
- _tablet_schema->append_column(TabletColumn(column_desc));
- }
- }
- {
- std::shared_lock rdlock(_tablet->get_header_lock());
- const RowsetSharedPtr rowset = _tablet->rowset_with_max_version();
- if (rowset == nullptr) {
- std::stringstream ss;
- ss << "fail to get latest version of tablet: " << tablet_id;
- LOG(WARNING) << ss.str();
- return Status::InternalError(ss.str());
- }
-
- // acquire tablet rowset readers at the beginning of the scan node
- // to prevent this case: when there are lots of olap scanners to run for example 10000
- // the rowsets maybe compacted when the last olap scanner starts
- Version rd_version(0, _version);
- Status acquire_reader_st =
- _tablet->capture_rs_readers(rd_version, &_tablet_reader_params.rs_readers);
- if (!acquire_reader_st.ok()) {
- LOG(WARNING) << "fail to init reader.res=" << acquire_reader_st;
- std::stringstream ss;
- ss << "failed to initialize storage reader. tablet=" << _tablet->full_name()
- << ", res=" << acquire_reader_st
- << ", backend=" << BackendOptions::get_localhost();
- return Status::InternalError(ss.str());
- }
-
- // Initialize tablet_reader_params
- RETURN_IF_ERROR(_init_tablet_reader_params(key_ranges, filters, bloom_filters,
- function_filters));
- }
- }
-
- return Status::OK();
-}
-
-Status VOlapScanner::open() {
- SCOPED_TIMER(_parent->_reader_init_timer);
-
- _runtime_filter_marks.resize(_parent->runtime_filter_descs().size(), false);
-
- auto res = _tablet_reader->init(_tablet_reader_params);
- if (!res.ok()) {
- std::stringstream ss;
- ss << "failed to initialize storage reader. tablet="
- << _tablet_reader_params.tablet->full_name() << ", res=" << res
- << ", backend=" << BackendOptions::get_localhost();
- return Status::InternalError(ss.str());
- }
- return Status::OK();
-}
-
-TabletStorageType VOlapScanner::get_storage_type() {
- int local_reader = 0;
- for (const auto& reader : _tablet_reader_params.rs_readers) {
- if (reader->rowset()->rowset_meta()->resource_id().empty()) {
- local_reader++;
- }
- }
- int total_reader = _tablet_reader_params.rs_readers.size();
-
- if (local_reader == total_reader) {
- return TabletStorageType::STORAGE_TYPE_LOCAL;
- } else if (local_reader == 0) {
- return TabletStorageType::STORAGE_TYPE_REMOTE;
- }
- return TabletStorageType::STORAGE_TYPE_REMOTE_AND_LOCAL;
-}
-
-// it will be called under tablet read lock because capture rs readers need
-Status VOlapScanner::_init_tablet_reader_params(
- const std::vector<OlapScanRange*>& key_ranges, const std::vector<TCondition>& filters,
- const std::vector<std::pair<string, std::shared_ptr<BloomFilterFuncBase>>>& bloom_filters,
- const std::vector<FunctionFilter>& function_filters) {
- // if the table with rowset [0-x] or [0-1] [2-y], and [0-1] is empty
- bool single_version =
- (_tablet_reader_params.rs_readers.size() == 1 &&
- _tablet_reader_params.rs_readers[0]->rowset()->start_version() == 0 &&
- !_tablet_reader_params.rs_readers[0]
- ->rowset()
- ->rowset_meta()
- ->is_segments_overlapping()) ||
- (_tablet_reader_params.rs_readers.size() == 2 &&
- _tablet_reader_params.rs_readers[0]->rowset()->rowset_meta()->num_rows() == 0 &&
- _tablet_reader_params.rs_readers[1]->rowset()->start_version() == 2 &&
- !_tablet_reader_params.rs_readers[1]
- ->rowset()
- ->rowset_meta()
- ->is_segments_overlapping());
-
- if (_runtime_state->skip_storage_engine_merge()) {
- _tablet_reader_params.direct_mode = true;
- _aggregation = true;
- } else {
- _tablet_reader_params.direct_mode = _aggregation || single_version ||
- _parent->_olap_scan_node.__isset.push_down_agg_type_opt;
- }
- RETURN_IF_ERROR(_init_return_columns());
-
- _tablet_reader_params.tablet = _tablet;
- _tablet_reader_params.tablet_schema = _tablet_schema;
- _tablet_reader_params.reader_type = READER_QUERY;
- _tablet_reader_params.aggregation = _aggregation;
- if (_parent->_olap_scan_node.__isset.push_down_agg_type_opt)
- _tablet_reader_params.push_down_agg_type_opt =
- _parent->_olap_scan_node.push_down_agg_type_opt;
- _tablet_reader_params.version = Version(0, _version);
-
- // Condition
- for (auto& filter : filters) {
- _tablet_reader_params.conditions.push_back(filter);
- }
- std::copy(bloom_filters.cbegin(), bloom_filters.cend(),
- std::inserter(_tablet_reader_params.bloom_filters,
- _tablet_reader_params.bloom_filters.begin()));
-
- std::copy(function_filters.cbegin(), function_filters.cend(),
- std::inserter(_tablet_reader_params.function_filters,
- _tablet_reader_params.function_filters.begin()));
- if (!_runtime_state->skip_delete_predicate()) {
- auto& delete_preds = _tablet->delete_predicates();
- std::copy(delete_preds.cbegin(), delete_preds.cend(),
- std::inserter(_tablet_reader_params.delete_predicates,
- _tablet_reader_params.delete_predicates.begin()));
- }
-
- // Merge the columns in delete predicate that not in latest schema in to current tablet schema
- for (auto& del_pred_rs : _tablet_reader_params.delete_predicates) {
- _tablet_schema->merge_dropped_columns(_tablet->tablet_schema(del_pred_rs->version()));
- }
-
- // Range
- for (auto key_range : key_ranges) {
- if (key_range->begin_scan_range.size() == 1 &&
- key_range->begin_scan_range.get_value(0) == NEGATIVE_INFINITY) {
- continue;
- }
-
- _tablet_reader_params.start_key_include = key_range->begin_include;
- _tablet_reader_params.end_key_include = key_range->end_include;
-
- _tablet_reader_params.start_key.push_back(key_range->begin_scan_range);
- _tablet_reader_params.end_key.push_back(key_range->end_scan_range);
- }
-
- _tablet_reader_params.profile = _parent->runtime_profile();
- _tablet_reader_params.runtime_state = _runtime_state;
-
- _tablet_reader_params.origin_return_columns = &_return_columns;
- _tablet_reader_params.tablet_columns_convert_to_null_set = &_tablet_columns_convert_to_null_set;
-
- if (_tablet_reader_params.direct_mode) {
- _tablet_reader_params.return_columns = _return_columns;
- } else {
- // we need to fetch all key columns to do the right aggregation on storage engine side.
- for (size_t i = 0; i < _tablet_schema->num_key_columns(); ++i) {
- _tablet_reader_params.return_columns.push_back(i);
- }
- for (auto index : _return_columns) {
- if (_tablet_schema->column(index).is_key()) {
- continue;
- } else {
- _tablet_reader_params.return_columns.push_back(index);
- }
- }
- // expand the sequence column
- if (_tablet_schema->has_sequence_col()) {
- bool has_replace_col = false;
- for (auto col : _return_columns) {
- if (_tablet_schema->column(col).aggregation() ==
- FieldAggregationMethod::OLAP_FIELD_AGGREGATION_REPLACE) {
- has_replace_col = true;
- break;
- }
- }
- if (auto sequence_col_idx = _tablet_schema->sequence_col_idx();
- has_replace_col && std::find(_return_columns.begin(), _return_columns.end(),
- sequence_col_idx) == _return_columns.end()) {
- _tablet_reader_params.return_columns.push_back(sequence_col_idx);
- }
- }
- }
-
- // If a agg node is this scan node direct parent
- // we will not call agg object finalize method in scan node,
- // to avoid the unnecessary SerDe and improve query performance
- _tablet_reader_params.need_agg_finalize = _need_agg_finalize;
-
- if (!config::disable_storage_page_cache) {
- _tablet_reader_params.use_page_cache = true;
- }
-
- if (_tablet->enable_unique_key_merge_on_write()) {
- _tablet_reader_params.delete_bitmap = &_tablet->tablet_meta()->delete_bitmap();
- }
-
- if (!_runtime_state->skip_storage_engine_merge()) {
- if (_parent->_olap_scan_node.__isset.sort_info &&
- _parent->_olap_scan_node.sort_info.is_asc_order.size() > 0) {
- _limit = _parent->_limit_per_scanner;
- _tablet_reader_params.read_orderby_key = true;
- if (!_parent->_olap_scan_node.sort_info.is_asc_order[0]) {
- _tablet_reader_params.read_orderby_key_reverse = true;
- }
- _tablet_reader_params.read_orderby_key_num_prefix_columns =
- _parent->_olap_scan_node.sort_info.is_asc_order.size();
- }
- }
-
- return Status::OK();
-}
-
-Status VOlapScanner::_init_return_columns() {
- for (auto slot : _tuple_desc->slots()) {
- if (!slot->is_materialized()) {
- continue;
- }
-
- int32_t index = slot->col_unique_id() >= 0
- ? _tablet_schema->field_index(slot->col_unique_id())
- : _tablet_schema->field_index(slot->col_name());
- if (index < 0) {
- std::stringstream ss;
- ss << "field name is invalid. field=" << slot->col_name();
- LOG(WARNING) << ss.str();
- return Status::InternalError(ss.str());
- }
- _return_columns.push_back(index);
- if (slot->is_nullable() && !_tablet_schema->column(index).is_nullable()) {
- _tablet_columns_convert_to_null_set.emplace(index);
- }
- }
-
- if (_return_columns.empty()) {
- return Status::InternalError("failed to build storage scanner, no materialized slot!");
- }
- return Status::OK();
-}
-
-Status VOlapScanner::get_block(RuntimeState* state, vectorized::Block* block, bool* eof) {
- // only empty block should be here
- DCHECK(block->rows() == 0);
-
- int64_t raw_rows_threshold = raw_rows_read() + config::doris_scanner_row_num;
- if (!block->mem_reuse()) {
- for (const auto slot_desc : _tuple_desc->slots()) {
- block->insert(ColumnWithTypeAndName(slot_desc->get_empty_mutable_column(),
- slot_desc->get_data_type_ptr(),
- slot_desc->col_name()));
- }
- }
-
- {
- SCOPED_TIMER(_parent->_scan_timer);
- do {
- // Read one block from block reader
- auto res = _tablet_reader->next_block_with_aggregation(block, nullptr, nullptr, eof);
- if (!res) {
- std::stringstream ss;
- ss << "Internal Error: read storage fail. res=" << res
- << ", tablet=" << _tablet->full_name()
- << ", backend=" << BackendOptions::get_localhost();
- return res;
- }
- _num_rows_read += block->rows();
- _update_realtime_counter();
- RETURN_IF_ERROR(
- VExprContext::filter_block(_vconjunct_ctx, block, _tuple_desc->slots().size()));
- // record rows return (after filter) for _limit check
- _num_rows_return += block->rows();
- } while (block->rows() == 0 && !(*eof) && raw_rows_read() < raw_rows_threshold);
- }
- // NOTE:
- // There is no need to check raw_bytes_threshold since block->rows() == 0 is checked first.
- // But checking raw_bytes_threshold is still added here for consistency with raw_rows_threshold
- // and olap_scanner.cpp.
-
- // set eof to true if per scanner limit is reached
- // currently for query: ORDER BY key LIMIT n
- if (_limit > 0 && _num_rows_return > _limit) {
- *eof = true;
- }
-
- return Status::OK();
-}
-
-void VOlapScanner::_update_realtime_counter() {
- auto& stats = _tablet_reader->stats();
- COUNTER_UPDATE(_parent->_read_compressed_counter, stats.compressed_bytes_read);
- _compressed_bytes_read += stats.compressed_bytes_read;
- _tablet_reader->mutable_stats()->compressed_bytes_read = 0;
-
- COUNTER_UPDATE(_parent->_raw_rows_counter, stats.raw_rows_read);
- // if raw_rows_read is reset, scanNode will scan all table rows which may cause BE crash
- _raw_rows_read += stats.raw_rows_read;
- _tablet_reader->mutable_stats()->raw_rows_read = 0;
-}
-
-Status VOlapScanner::close(RuntimeState* state) {
- if (_is_closed) {
- return Status::OK();
- }
- for (auto& ctx : _stale_vexpr_ctxs) {
- ctx->close(state);
- }
- if (_vconjunct_ctx) {
- _vconjunct_ctx->close(state);
- }
- // olap scan node will call scanner.close() when finished
- // will release resources here
- // if not clear rowset readers in read_params here
- // readers will be release when runtime state deconstructed but
- // deconstructor in reader references runtime state
- // so that it will core
- _tablet_reader_params.rs_readers.clear();
- update_counter();
- _tablet_reader.reset();
- _is_closed = true;
- return Status::OK();
-}
-
-void VOlapScanner::update_counter() {
- if (_has_update_counter) {
- return;
- }
- auto& stats = _tablet_reader->stats();
-
- COUNTER_UPDATE(_parent->rows_read_counter(), _num_rows_read);
- COUNTER_UPDATE(_parent->_rows_pushed_cond_filtered_counter, _num_rows_pushed_cond_filtered);
-
- COUNTER_UPDATE(_parent->_io_timer, stats.io_ns);
- COUNTER_UPDATE(_parent->_read_compressed_counter, stats.compressed_bytes_read);
- _compressed_bytes_read += stats.compressed_bytes_read;
- COUNTER_UPDATE(_parent->_decompressor_timer, stats.decompress_ns);
- COUNTER_UPDATE(_parent->_read_uncompressed_counter, stats.uncompressed_bytes_read);
- COUNTER_UPDATE(_parent->bytes_read_counter(), stats.bytes_read);
-
- COUNTER_UPDATE(_parent->_block_load_timer, stats.block_load_ns);
- COUNTER_UPDATE(_parent->_block_load_counter, stats.blocks_load);
- COUNTER_UPDATE(_parent->_block_fetch_timer, stats.block_fetch_ns);
- COUNTER_UPDATE(_parent->_block_seek_timer, stats.block_seek_ns);
- COUNTER_UPDATE(_parent->_block_seek_counter, stats.block_seek_num);
- COUNTER_UPDATE(_parent->_block_convert_timer, stats.block_convert_ns);
-
- COUNTER_UPDATE(_parent->_raw_rows_counter, stats.raw_rows_read);
- // if raw_rows_read is reset, scanNode will scan all table rows which may cause BE crash
- _raw_rows_read += _tablet_reader->mutable_stats()->raw_rows_read;
- // COUNTER_UPDATE(_parent->_filtered_rows_counter, stats.num_rows_filtered);
- COUNTER_UPDATE(_parent->_vec_cond_timer, stats.vec_cond_ns);
- COUNTER_UPDATE(_parent->_short_cond_timer, stats.short_cond_ns);
- COUNTER_UPDATE(_parent->_block_init_timer, stats.block_init_ns);
- COUNTER_UPDATE(_parent->_block_init_seek_timer, stats.block_init_seek_ns);
- COUNTER_UPDATE(_parent->_block_init_seek_counter, stats.block_init_seek_num);
- COUNTER_UPDATE(_parent->_first_read_timer, stats.first_read_ns);
- COUNTER_UPDATE(_parent->_first_read_seek_timer, stats.block_first_read_seek_ns);
- COUNTER_UPDATE(_parent->_first_read_seek_counter, stats.block_first_read_seek_num);
- COUNTER_UPDATE(_parent->_lazy_read_timer, stats.lazy_read_ns);
- COUNTER_UPDATE(_parent->_lazy_read_seek_timer, stats.block_lazy_read_seek_ns);
- COUNTER_UPDATE(_parent->_lazy_read_seek_counter, stats.block_lazy_read_seek_num);
- COUNTER_UPDATE(_parent->_output_col_timer, stats.output_col_ns);
- COUNTER_UPDATE(_parent->_rows_vec_cond_counter, stats.rows_vec_cond_filtered);
-
- COUNTER_UPDATE(_parent->_stats_filtered_counter, stats.rows_stats_filtered);
- COUNTER_UPDATE(_parent->_bf_filtered_counter, stats.rows_bf_filtered);
- COUNTER_UPDATE(_parent->_del_filtered_counter, stats.rows_del_filtered);
- COUNTER_UPDATE(_parent->_del_filtered_counter, stats.rows_del_by_bitmap);
- COUNTER_UPDATE(_parent->_del_filtered_counter, stats.rows_vec_del_cond_filtered);
-
- COUNTER_UPDATE(_parent->_conditions_filtered_counter, stats.rows_conditions_filtered);
- COUNTER_UPDATE(_parent->_key_range_filtered_counter, stats.rows_key_range_filtered);
-
- COUNTER_UPDATE(_parent->_index_load_timer, stats.index_load_ns);
-
- size_t timer_count = sizeof(stats.general_debug_ns) / sizeof(*stats.general_debug_ns);
- for (size_t i = 0; i < timer_count; ++i) {
- COUNTER_UPDATE(_parent->_general_debug_timer[i], stats.general_debug_ns[i]);
- }
-
- COUNTER_UPDATE(_parent->_total_pages_num_counter, stats.total_pages_num);
- COUNTER_UPDATE(_parent->_cached_pages_num_counter, stats.cached_pages_num);
-
- COUNTER_UPDATE(_parent->_bitmap_index_filter_counter, stats.rows_bitmap_index_filtered);
- COUNTER_UPDATE(_parent->_bitmap_index_filter_timer, stats.bitmap_index_filter_timer);
-
- COUNTER_UPDATE(_parent->_filtered_segment_counter, stats.filtered_segment_number);
- COUNTER_UPDATE(_parent->_total_segment_counter, stats.total_segment_number);
-
- DorisMetrics::instance()->query_scan_bytes->increment(_compressed_bytes_read);
- DorisMetrics::instance()->query_scan_rows->increment(_raw_rows_read);
-
- _tablet->query_scan_bytes->increment(_compressed_bytes_read);
- _tablet->query_scan_rows->increment(_raw_rows_read);
- _tablet->query_scan_count->increment(1);
-
- _has_update_counter = true;
-}
-} // namespace doris::vectorized
diff --git a/be/src/vec/exec/volap_scanner.h b/be/src/vec/exec/volap_scanner.h
deleted file mode 100644
index e5dec37469..0000000000
--- a/be/src/vec/exec/volap_scanner.h
+++ /dev/null
@@ -1,153 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-#pragma once
-
-#include "exec/exec_node.h"
-#include "exec/olap_utils.h"
-#include "gen_cpp/PaloInternalService_types.h"
-#include "gen_cpp/PlanNodes_types.h"
-#include "olap/tablet.h"
-#include "vec/exprs/vexpr_context.h"
-#include "vec/olap/block_reader.h"
-
-namespace doris {
-class RuntimeProfile;
-class Field;
-class RowBatch;
-
-namespace vectorized {
-class VOlapScanNode;
-
-class VOlapScanner {
-public:
- VOlapScanner(RuntimeState* runtime_state, VOlapScanNode* parent, bool aggregation,
- bool need_agg_finalize, const TPaloScanRange& scan_range);
- virtual ~VOlapScanner() = default;
-
- Status prepare(const TPaloScanRange& scan_range, const std::vector<OlapScanRange*>& key_ranges,
- const std::vector<TCondition>& filters,
- const std::vector<std::pair<std::string, std::shared_ptr<BloomFilterFuncBase>>>&
- bloom_filters,
- const std::vector<FunctionFilter>& function_filters);
-
- Status open();
-
- Status get_block(RuntimeState* state, vectorized::Block* block, bool* eof);
-
- Status close(RuntimeState* state);
-
- Status get_batch(RuntimeState* state, RowBatch* row_batch, bool* eos) {
- return Status::NotSupported("Not Implemented VOlapScanNode Node::get_next scalar");
- }
-
- RuntimeState* runtime_state() { return _runtime_state; }
-
- VExprContext** vconjunct_ctx_ptr() { return &_vconjunct_ctx; }
-
- void discard_conjuncts() {
- _vconjunct_ctx->mark_as_stale();
- _stale_vexpr_ctxs.push_back(_vconjunct_ctx);
- _vconjunct_ctx = nullptr;
- }
-
- void mark_to_need_to_close() { _need_to_close = true; }
-
- bool need_to_close() { return _need_to_close; }
-
- int id() const { return _id; }
- void set_id(int id) { _id = id; }
- bool is_open() const { return _is_open; }
- void set_opened() { _is_open = true; }
-
- int64_t raw_rows_read() const { return _raw_rows_read; }
-
- void update_counter();
-
- const std::string& scan_disk() const { return _tablet->data_dir()->path(); }
-
- void start_wait_worker_timer() {
- _watcher.reset();
- _watcher.start();
- }
-
- int64_t update_wait_worker_timer() const { return _watcher.elapsed_time(); }
-
- std::vector<bool>* mutable_runtime_filter_marks() { return &_runtime_filter_marks; }
-
- TabletStorageType get_storage_type();
-
-private:
- Status _init_tablet_reader_params(
- const std::vector<OlapScanRange*>& key_ranges, const std::vector<TCondition>& filters,
- const std::vector<std::pair<string, std::shared_ptr<BloomFilterFuncBase>>>&
- bloom_filters,
- const std::vector<FunctionFilter>& function_filters);
- Status _init_return_columns();
-
- // Update profile that need to be reported in realtime.
- void _update_realtime_counter();
-
- virtual void set_tablet_reader() { _tablet_reader = std::make_unique<BlockReader>(); }
-
- RuntimeState* _runtime_state;
- VOlapScanNode* _parent;
- const TupleDescriptor* _tuple_desc; /**< tuple descriptor */
-
- // to record which runtime filters have been used
- std::vector<bool> _runtime_filter_marks;
-
- int64_t _limit = -1;
-
- int _id;
- bool _is_open;
- bool _aggregation;
- bool _need_agg_finalize = true;
- bool _has_update_counter = false;
-
- TabletReader::ReaderParams _tablet_reader_params;
- std::unique_ptr<TabletReader> _tablet_reader;
-
- TabletSharedPtr _tablet;
- int64_t _version;
-
- std::vector<uint32_t> _return_columns;
- std::unordered_set<uint32_t> _tablet_columns_convert_to_null_set;
-
- // time costed and row returned statistics
- int64_t _num_rows_read = 0;
- int64_t _raw_rows_read = 0;
- int64_t _num_rows_return = 0;
- int64_t _compressed_bytes_read = 0;
-
- // number rows filtered by pushed condition
- int64_t _num_rows_pushed_cond_filtered = 0;
-
- bool _is_closed = false;
-
- MonotonicStopWatch _watcher;
-
- VExprContext* _vconjunct_ctx = nullptr;
- bool _need_to_close = false;
-
- TabletSchemaSPtr _tablet_schema;
-
- std::vector<VExprContext*> _stale_vexpr_ctxs;
-};
-
-} // namespace vectorized
-} // namespace doris
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 07/39: [feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF (#13994)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2e0af89b7e81c99b1c89610eede505c2e87e3a93
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Tue Nov 8 14:02:41 2022 +0800
[feature](multi-catalog) Support data on s3-compatible oss and support aliyun DLF (#13994)
Support Aliyun DLF
Support data on s3-compatible object storage, such as aliyun oss.
Refactor some interface of catalog, to make it more tidy.
Fix bug that the default text format field delimiter of hive should be \x01
Add a new class PooledHiveMetaStoreClient to wrap the IMetaStoreClient.
---
.asf.yaml | 1 +
build.sh | 17 ++-
.../docs/ecosystem/external-table/multi-catalog.md | 68 +++++++++-
.../Create/CREATE-TABLE-AS-SELECT.md | 6 +
.../docs/ecosystem/external-table/multi-catalog.md | 71 +++++++++-
.../Create/CREATE-TABLE-AS-SELECT.md | 6 +
.../org/apache/doris/analysis/CreateTableStmt.java | 2 +-
.../doris/catalog/HiveMetaStoreClientHelper.java | 92 ++++++-------
.../doris/catalog/external/HMSExternalTable.java | 46 ++-----
.../java/org/apache/doris/common/FeConstants.java | 10 ++
.../org/apache/doris/datasource/CatalogMgr.java | 2 +-
.../apache/doris/datasource/EsExternalCatalog.java | 31 +----
.../apache/doris/datasource/ExternalCatalog.java | 48 ++++++-
.../doris/datasource/HMSExternalCatalog.java | 83 +++---------
.../apache/doris/datasource/InternalCatalog.java | 35 +++--
.../datasource/PooledHiveMetaStoreClient.java | 146 +++++++++++++++++++++
.../doris/planner/external/HiveScanProvider.java | 49 +++++--
.../doris/planner/external/QueryScanProvider.java | 3 +
18 files changed, 506 insertions(+), 210 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 532061b89d..e83a5f1421 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -58,6 +58,7 @@ github:
- Build Broker
- Build Documents
- BE UT (Clang)
+ - ShellCheck
required_pull_request_reviews:
dismiss_stale_reviews: true
diff --git a/build.sh b/build.sh
index 65559f026e..aba8a7f565 100755
--- a/build.sh
+++ b/build.sh
@@ -50,9 +50,10 @@ Usage: $0 <options>
-j build Backend parallel
Environment variables:
- USE_AVX2 If the CPU does not support AVX2 instruction set, please set USE_AVX2=0. Default is ON.
- STRIP_DEBUG_INFO If set STRIP_DEBUG_INFO=ON, the debug information in the compiled binaries will be stored separately in the 'be/lib/debug_info' directory. Default is OFF.
- DISABLE_JAVA_UDF If set DISABLE_JAVA_UDF=ON, we will do not build binary with java-udf. Default is OFF.
+ USE_AVX2 If the CPU does not support AVX2 instruction set, please set USE_AVX2=0. Default is ON.
+ STRIP_DEBUG_INFO If set STRIP_DEBUG_INFO=ON, the debug information in the compiled binaries will be stored separately in the 'be/lib/debug_info' directory. Default is OFF.
+ DISABLE_JAVA_UDF If set DISABLE_JAVA_UDF=ON, we will do not build binary with java-udf. Default is OFF.
+ DISABLE_JAVA_CHECK_STYLE If set DISABLE_JAVA_CHECK_STYLE=ON, it will skip style check of java code in FE.
Eg.
$0 build all
$0 --be build Backend
@@ -289,6 +290,10 @@ if [[ -z "${DISABLE_JAVA_UDF}" ]]; then
DISABLE_JAVA_UDF='OFF'
fi
+if [[ -z "${DISABLE_JAVA_CHECK_STYLE}" ]]; then
+ DISABLE_JAVA_CHECK_STYLE='OFF'
+fi
+
if [[ -z "${RECORD_COMPILER_SWITCHES}" ]]; then
RECORD_COMPILER_SWITCHES='OFF'
fi
@@ -463,7 +468,11 @@ if [[ "${FE_MODULES}" != '' ]]; then
if [[ "${CLEAN}" -eq 1 ]]; then
clean_fe
fi
- "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -DskipTests
+ if [[ "${DISABLE_JAVA_CHECK_STYLE}" = "ON" ]]; then
+ "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -DskipTests -Dcheckstyle.skip=true
+ else
+ "${MVN_CMD}" package -pl ${FE_MODULES:+${FE_MODULES}} -DskipTests
+ fi
cd "${DORIS_HOME}"
fi
diff --git a/docs/en/docs/ecosystem/external-table/multi-catalog.md b/docs/en/docs/ecosystem/external-table/multi-catalog.md
index 7cb081a6d7..905993f608 100644
--- a/docs/en/docs/ecosystem/external-table/multi-catalog.md
+++ b/docs/en/docs/ecosystem/external-table/multi-catalog.md
@@ -24,6 +24,8 @@ specific language governing permissions and limitations
under the License.
-->
+<version since="1.2.0">
+
# Multi-Catalog
Multi-Catalog is a feature introduced in Doris 1.2.0, which aims to make it easier to interface with external data sources to enhance Doris' data lake analysis and federated data query capabilities.
@@ -292,7 +294,7 @@ mysql> select * from test;
+------------+-------------+--------+-------+
```
-## Parameters that:
+#### Parameters:
Parameter | Description
---|---
@@ -304,6 +306,68 @@ Parameter | Description
**elasticsearch.nodes_discovery** | Whether or not to enable ES node discovery, the default is true. In network isolation, set this parameter to false. Only the specified node is connected.
**elasticsearch.ssl** | Whether ES cluster enables https access mode, the current FE/BE implementation is to trust all
+### Connect Aliyun Data Lake Formation
+
+> [What is Data Lake Formation](https://www.alibabacloud.com/product/datalake-formation)
+
+1. Create hive-site.xml
+
+ Create hive-site.xml and put it in `fe/conf` and `be/conf`.
+
+ ```
+ <?xml version="1.0"?>
+ <configuration>
+ <!--Set to use dlf client-->
+ <property>
+ <name>hive.metastore.type</name>
+ <value>dlf</value>
+ </property>
+ <property>
+ <name>dlf.catalog.endpoint</name>
+ <value>dlf-vpc.cn-beijing.aliyuncs.com</value>
+ </property>
+ <property>
+ <name>dlf.catalog.region</name>
+ <value>cn-beijing</value>
+ </property>
+ <property>
+ <name>dlf.catalog.proxyMode</name>
+ <value>DLF_ONLY</value>
+ </property>
+ <property>
+ <name>dlf.catalog.uid</name>
+ <value>20000000000000000</value>
+ </property>
+ <property>
+ <name>dlf.catalog.accessKeyId</name>
+ <value>XXXXXXXXXXXXXXX</value>
+ </property>
+ <property>
+ <name>dlf.catalog.accessKeySecret</name>
+ <value>XXXXXXXXXXXXXXXXX</value>
+ </property>
+ </configuration>
+ ```
+
+ * `dlf.catalog.endpoint`: DLF Endpoint. See: [Regions and endpoints of DLF](https://www.alibabacloud.com/help/en/data-lake-formation/latest/regions-and-endpoints)
+ * `dlf.catalog.region`: DLF Regio. See: [Regions and endpoints of DLF](https://www.alibabacloud.com/help/en/data-lake-formation/latest/regions-and-endpoints)
+ * `dlf.catalog.uid`: Ali Cloud Account ID. That is, the "cloud account ID" of the personal information in the upper right corner of the Alibaba Cloud console. * `dlf.catalog.accessKeyId`: AccessKey. See: [Ali Could Console](https://ram.console.aliyun.com/manage/ak).
+ * `dlf.catalog.accessKeySecret`: SecretKey. See: [Ali Could Console](https://ram.console.aliyun.com/manage/ak).
+
+ Other configuration items are fixed values and do not need to be changed.
+
+2. Restart FE and create a catalog with the `CREATE CATALOG` statement.
+
+ ```
+ CREATE CATALOG dlf PROPERTIES (
+ "type"="hms",
+ "hive.metastore.uris" = "thrift://127.0.0.1:9083"
+ );
+ ```
+
+ where `type` is fixed to `hms`. The value of `hive.metastore.uris` can be filled in at will, but it will not be used in practice. But it needs to be filled in the standard hive metastore thrift uri format.
+
+After that, the metadata under DLF can be accessed like a normal Hive MetaStore.
## Column Type Mapping
@@ -366,3 +430,5 @@ Metadata changes of external data sources, such as creating, dropping tables, ad
Currently, users need to manually refresh metadata via the [REFRESH CATALOG](../../sql-manual/sql-reference/Utility-Statements/REFRESH-CATALOG.md) command.
Automatic synchronization of metadata will be supported soon.
+
+</version>
\ No newline at end of file
diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md
index 7498fdaa98..9589e6a33b 100644
--- a/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md
+++ b/docs/en/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md
@@ -55,6 +55,12 @@ illustrate:
- After a table is created, data is imported. If the import fails, the table is deleted
- You can specify the key type. The default key type is `Duplicate Key`
+<version since='1.2'>
+
+- If the created source is an external table and the first column is of type String, the first column is automatically set to VARCHAR(65533). Because of Doris internal table, String column is not allowed as first column.
+
+</version>
+
### Example
1. Using the field names in the SELECT statement
diff --git a/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md b/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
index 5b79d51821..0d46d212d6 100644
--- a/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
+++ b/docs/zh-CN/docs/ecosystem/external-table/multi-catalog.md
@@ -24,6 +24,8 @@ specific language governing permissions and limitations
under the License.
-->
+<version since="1.2.0">
+
# 多源数据目录
多源数据目录(Multi-Catalog)是 Doris 1.2.0 版本中推出的功能,旨在能够更方便对接外部数据目录,以增强Doris的数据湖分析和联邦数据查询能力。
@@ -292,7 +294,7 @@ mysql> select * from test;
+------------+-------------+--------+-------+
```
-## 参数说明:
+#### 参数说明:
参数 | 说明
---|---
@@ -304,6 +306,71 @@ mysql> select * from test;
**elasticsearch.nodes_discovery** | 是否开启 ES 节点发现,默认为 true,在网络隔离环境下设置为 false,只连接指定节点
**elasticsearch.ssl** | ES 是否开启 https 访问模式,目前在 fe/be 实现方式为信任所有
+### 连接阿里云 Data Lake Formation
+
+> [什么是 Data Lake Formation](https://www.aliyun.com/product/bigdata/dlf)
+
+1. 创建 hive-site.xml
+
+ 创建 hive-site.xml 文件,并将其放置在 `fe/conf` 和 `be/conf` 目录下。
+
+ ```
+ <?xml version="1.0"?>
+ <configuration>
+ <!--Set to use dlf client-->
+ <property>
+ <name>hive.metastore.type</name>
+ <value>dlf</value>
+ </property>
+ <property>
+ <name>dlf.catalog.endpoint</name>
+ <value>dlf-vpc.cn-beijing.aliyuncs.com</value>
+ </property>
+ <property>
+ <name>dlf.catalog.region</name>
+ <value>cn-beijing</value>
+ </property>
+ <property>
+ <name>dlf.catalog.proxyMode</name>
+ <value>DLF_ONLY</value>
+ </property>
+ <property>
+ <name>dlf.catalog.uid</name>
+ <value>20000000000000000</value>
+ </property>
+ <property>
+ <name>dlf.catalog.accessKeyId</name>
+ <value>XXXXXXXXXXXXXXX</value>
+ </property>
+ <property>
+ <name>dlf.catalog.accessKeySecret</name>
+ <value>XXXXXXXXXXXXXXXXX</value>
+ </property>
+ </configuration>
+ ```
+
+ * `dlf.catalog.endpoint`:DLF Endpoint,参阅:[DLF Region和Endpoint对照表](https://www.alibabacloud.com/help/zh/data-lake-formation/latest/regions-and-endpoints)
+ * `dlf.catalog.region`:DLF Region,参阅:[DLF Region和Endpoint对照表](https://www.alibabacloud.com/help/zh/data-lake-formation/latest/regions-and-endpoints)
+ * `dlf.catalog.uid`:阿里云账号。即阿里云控制台右上角个人信息的“云账号ID”。
+ * `dlf.catalog.accessKeyId`:AccessKey。可以在 [阿里云控制台](https://ram.console.aliyun.com/manage/ak) 中创建和管理。
+ * `dlf.catalog.accessKeySecret`:SecretKey。可以在 [阿里云控制台](https://ram.console.aliyun.com/manage/ak) 中创建和管理。
+
+ 其他配置项为固定值,无需改动。
+
+2. 重启 FE,并通过 `CREATE CATALOG` 语句创建 catalog。
+
+ ```
+ CREATE CATALOG dlf PROPERTIES (
+ "type"="hms",
+ "hive.metastore.uris" = "thrift://127.0.0.1:9083"
+ );
+ ```
+
+ 其中 `type` 固定为 `hms`。 `hive.metastore.uris` 的值随意填写即可,实际不会使用。但需要按照标准 hive metastore thrift uri 格式填写。
+
+之后,可以像正常的 Hive MetaStore 一样,访问 DLF 下的元数据。
+
+
## 列类型映射
用户创建 Catalog 后,Doris 会自动同步数据目录的数据库和表,针对不同的数据目录和数据表格式,Doris 会进行以下列映射关系。
@@ -365,3 +432,5 @@ Doris 的权限管理功能提供了对 Cataloig 层级的扩展,具体可参
目前需要用户通过 [REFRESH CATALOG](../../sql-manual/sql-reference/Utility-Statements/REFRESH-CATALOG.md) 命令手动刷新元数据。
后续会支持元数据的自动同步。
+
+</version>
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md
index 82938f9315..f2f3f3a834 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE-AS-SELECT.md
@@ -55,6 +55,12 @@ CREATE TABLE table_name [( column_name_list )]
- 创建表成功后,会进行数据导入,如果导入失败,将会删除表
- 可以自行指定 key type,默认为`Duplicate Key`
+<version since='1.2'>
+
+- 如果创建的来源为外部表,并且第一列为 String 类型,则会自动将第一列设置为 VARCHAR(65533)。因为 Doris 内部表,不允许 String 列作为第一列。
+
+</version>
+
### Example
1. 使用 select 语句中的字段名
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index bf3b06783c..7ba06377a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -332,7 +332,7 @@ public class CreateTableStmt extends DdlStmt {
}
keysColumnNames.add(columnDef.getName());
}
- // The OLAP table must has at least one short key and the float and double should not be short key.
+ // The OLAP table must have at least one short key and the float and double should not be short key.
// So the float and double could not be the first column in OLAP table.
if (keysColumnNames.isEmpty()) {
throw new AnalysisException("The olap table first column could not be float, double, string"
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
index dbc402b10d..8440b5e6a2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
@@ -39,6 +39,7 @@ import org.apache.doris.thrift.TExprOpcode;
import com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient;
import com.google.common.base.Strings;
+import com.google.common.collect.Maps;
import com.google.common.collect.Queues;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
@@ -91,7 +92,10 @@ import java.util.stream.Collectors;
*/
public class HiveMetaStoreClientHelper {
private static final Logger LOG = LogManager.getLogger(HiveMetaStoreClientHelper.class);
+
+ public static final String HIVE_METASTORE_URIS = "hive.metastore.uris";
public static final String HIVE_METASTORE_TYPE = "hive.metastore.type";
+ public static final String DLF_TYPE = "dlf";
private static final Pattern digitPattern = Pattern.compile("(\\d+)");
@@ -142,16 +146,12 @@ public class HiveMetaStoreClientHelper {
public static IMetaStoreClient getClient(String metaStoreUris) throws DdlException {
HiveConf hiveConf = new HiveConf();
hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, metaStoreUris);
- return getClient(hiveConf);
- }
-
- public static IMetaStoreClient getClient(HiveConf hiveConf) throws DdlException {
hiveConf.set(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
String.valueOf(Config.hive_metastore_client_timeout_second));
IMetaStoreClient metaStoreClient = null;
String type = hiveConf.get(HIVE_METASTORE_TYPE);
try {
- if (type.equalsIgnoreCase("dlf")) {
+ if ("dlf".equalsIgnoreCase(type)) {
// For aliyun DLF
metaStoreClient = new ProxyMetaStoreClient(hiveConf);
} else {
@@ -164,33 +164,6 @@ public class HiveMetaStoreClientHelper {
return metaStoreClient;
}
- /**
- * Check to see if the specified table exists in the specified database.
- *
- * @param client HiveMetaStoreClient
- * @param dbName the specified database name
- * @param tblName the specified table name
- * @return TRUE if specified.tableName exists, FALSE otherwise.
- * @throws DdlException
- */
- public static boolean tableExists(IMetaStoreClient client, String dbName, String tblName) throws DdlException {
- try {
- return client.tableExists(dbName, tblName);
- } catch (TException e) {
- LOG.warn("Hive metastore thrift exception: {}", e.getMessage());
- throw new DdlException("Connect hive metastore failed. Error: " + e.getMessage());
- } finally {
- dropClient(client);
- }
- }
-
- /**
- * close connection to meta store
- */
- public static void dropClient(IMetaStoreClient client) {
- client.close();
- }
-
/**
* Get data files of partitions in hive table, filter by partition predicate.
*
@@ -401,6 +374,7 @@ public class HiveMetaStoreClientHelper {
/**
* Get hive table with dbName and tableName.
+ * Only for Hudi.
*
* @param dbName database name
* @param tableName table name
@@ -408,6 +382,7 @@ public class HiveMetaStoreClientHelper {
* @return HiveTable
* @throws DdlException when get table from hive metastore failed.
*/
+ @Deprecated
public static Table getTable(String dbName, String tableName, String metaStoreUris) throws DdlException {
IMetaStoreClient client = getClient(metaStoreUris);
Table table;
@@ -422,26 +397,6 @@ public class HiveMetaStoreClientHelper {
return table;
}
- /**
- * Get table schema.
- *
- * @param dbName Database name.
- * @param tableName Table name.
- * @param metaStoreUris Hive metastore uri.
- */
- public static List<FieldSchema> getSchema(String dbName, String tableName, String metaStoreUris)
- throws DdlException {
- IMetaStoreClient client = getClient(metaStoreUris);
- try {
- return client.getSchema(dbName, tableName);
- } catch (TException e) {
- LOG.warn("Hive metastore thrift exception: {}", e.getMessage());
- throw new DdlException("Connect hive metastore failed. Error: " + e.getMessage());
- } finally {
- client.close();
- }
- }
-
/**
* Convert Doris expr to Hive expr, only for partition column
* @param tblName
@@ -927,6 +882,39 @@ public class HiveMetaStoreClientHelper {
}
return output.toString();
}
+
+ public static Map<String, String> getPropertiesForDLF(String catalogName, HiveConf hiveConf) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("get properties from hive-site.xml for catalog {}: {}", catalogName, hiveConf.getAllProperties());
+ }
+ Map<String, String> res = Maps.newHashMap();
+ String metastoreType = hiveConf.get(HIVE_METASTORE_TYPE);
+ if (!"dlf".equalsIgnoreCase(metastoreType)) {
+ return res;
+ }
+
+ // get following properties from hive-site.xml
+ // 1. region and endpoint. eg: cn-beijing
+ String region = hiveConf.get("dlf.catalog.region");
+ if (!Strings.isNullOrEmpty(region)) {
+ res.put(HiveTable.AWS_REGION, "oss-" + region);
+ res.put(HiveTable.S3_ENDPOINT, "http://oss-" + region + ".aliyuncs.com");
+ }
+
+ // 2. ak and sk
+ String ak = hiveConf.get("dlf.catalog.accessKeyId");
+ String sk = hiveConf.get("dlf.catalog.accessKeySecret");
+ if (!Strings.isNullOrEmpty(ak)) {
+ res.put(HiveTable.S3_AK, ak);
+ }
+ if (!Strings.isNullOrEmpty(sk)) {
+ res.put(HiveTable.S3_SK, sk);
+ }
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("get properties for oss in hive-site.xml for catalog {}: {}", catalogName, res);
+ }
+ return res;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalTable.java
index 3da7914ff8..3d8b2e5778 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/external/HMSExternalTable.java
@@ -25,6 +25,7 @@ import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.HMSExternalCatalog;
import org.apache.doris.datasource.InitTableLog;
+import org.apache.doris.datasource.PooledHiveMetaStoreClient;
import org.apache.doris.qe.MasterCatalogExecutor;
import org.apache.doris.thrift.THiveTable;
import org.apache.doris.thrift.TTableDescriptor;
@@ -32,16 +33,13 @@ import org.apache.doris.thrift.TTableType;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
-import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.metastore.api.Partition;
import org.apache.hadoop.hive.ql.exec.SerializationUtilities;
import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import org.apache.thrift.TException;
-import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -177,16 +175,12 @@ public class HMSExternalTable extends ExternalTable {
if (dlaType.equals(DLAType.UNKNOWN)) {
schemaChanged = true;
} else {
- try {
- for (FieldSchema field : HiveMetaStoreClientHelper.getSchema(dbName, name,
- ((HMSExternalCatalog) catalog).getHiveMetastoreUris())) {
- int columnId = (int) Env.getCurrentEnv().getNextId();
- tmpSchema.add(new Column(field.getName(),
- HiveMetaStoreClientHelper.hiveTypeToDorisType(field.getType()), true, null,
- true, null, field.getComment(), true, null, columnId));
- }
- } catch (DdlException e) {
- LOG.warn("Fail to get schema of hms table {}", name, e);
+ List<FieldSchema> schema = ((HMSExternalCatalog) catalog).getClient().getSchema(dbName, name);
+ for (FieldSchema field : schema) {
+ int columnId = (int) Env.getCurrentEnv().getNextId();
+ tmpSchema.add(new Column(field.getName(),
+ HiveMetaStoreClientHelper.hiveTypeToDorisType(field.getType()), true, null,
+ true, null, field.getComment(), true, null, columnId));
}
if (fullSchema == null || fullSchema.size() != tmpSchema.size()) {
schemaChanged = true;
@@ -219,17 +213,10 @@ public class HMSExternalTable extends ExternalTable {
if (remoteTable == null) {
synchronized (this) {
if (remoteTable == null) {
- String uri = ((HMSExternalCatalog) catalog).getHiveMetastoreUris();
- try {
- remoteTable = HiveMetaStoreClientHelper.getTable(dbName, name, uri);
- } catch (DdlException e) {
- LOG.warn("Fail to get remote hive table. db {}, table {}, uri {}", dbName, name, uri);
- throw new MetaNotFoundException(e);
- }
+ remoteTable = ((HMSExternalCatalog) catalog).getClient().getTable(dbName, name);
}
}
}
- // TODO: Refresh cached remoteTable
return remoteTable;
}
@@ -354,20 +341,11 @@ public class HMSExternalTable extends ExternalTable {
}
public List<Partition> getHivePartitions(ExprNodeGenericFuncDesc hivePartitionPredicate) throws DdlException {
- List<Partition> hivePartitions = new ArrayList<>();
- IMetaStoreClient client = ((HMSExternalCatalog) catalog).getClient();
- try {
- client.listPartitionsByExpr(remoteTable.getDbName(), remoteTable.getTableName(),
- SerializationUtilities.serializeExpressionToKryo(hivePartitionPredicate),
- null, (short) -1, hivePartitions);
- } catch (TException e) {
- LOG.warn("Hive metastore thrift exception: {}", e);
- throw new DdlException("Connect hive metastore failed: " + e.getMessage());
- } finally {
- client.close();
- }
+ List<Partition> hivePartitions = Lists.newArrayList();
+ PooledHiveMetaStoreClient client = ((HMSExternalCatalog) catalog).getClient();
+ client.listPartitionsByExpr(remoteTable.getDbName(), remoteTable.getTableName(),
+ SerializationUtilities.serializeExpressionToKryo(hivePartitionPredicate), hivePartitions);
return hivePartitions;
}
-
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java b/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
index 4cb2bd877b..6a9e07c804 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
@@ -64,4 +64,14 @@ public class FeConstants {
public static String csv_with_names_and_types = "csv_with_names_and_types";
public static String text = "text";
+
+ public static String FS_PREFIX_S3 = "s3";
+ public static String FS_PREFIX_S3A = "s3a";
+ public static String FS_PREFIX_S3N = "s3n";
+ public static String FS_PREFIX_OSS = "oss";
+ public static String FS_PREFIX_BOS = "bos";
+ public static String FS_PREFIX_COS = "cos";
+ public static String FS_PREFIX_OBS = "obs";
+ public static String FS_PREFIX_HDFS = "hdfs";
+ public static String FS_PREFIX_FILE = "file";
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
index b92a515751..e1687b0f7f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
@@ -104,7 +104,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
if (catalog != null) {
String catalogName = catalog.getName();
if (!catalogName.equals(InternalCatalog.INTERNAL_CATALOG_NAME)) {
- ((ExternalCatalog) catalog).setInitialized(false);
+ ((ExternalCatalog) catalog).setUninitialized();
}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
index e614c97006..29123395ed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/EsExternalCatalog.java
@@ -23,10 +23,8 @@ import org.apache.doris.catalog.external.EsExternalDatabase;
import org.apache.doris.catalog.external.ExternalDatabase;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.DdlException;
-import org.apache.doris.common.util.Util;
import org.apache.doris.external.elasticsearch.EsRestClient;
import org.apache.doris.external.elasticsearch.EsUtil;
-import org.apache.doris.qe.MasterCatalogExecutor;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
@@ -118,33 +116,13 @@ public class EsExternalCatalog extends ExternalCatalog {
}
}
- /**
- * Datasource can't be init when creating because the external datasource may depend on third system.
- * So you have to make sure the client of third system is initialized before any method was called.
- */
@Override
- public synchronized void makeSureInitialized() {
- if (!objectCreated) {
- esRestClient = new EsRestClient(this.nodes, this.username, this.password, this.enableSsl);
- objectCreated = true;
- }
- if (!initialized) {
- if (!Env.getCurrentEnv().isMaster()) {
- // Forward to master and wait the journal to replay.
- MasterCatalogExecutor remoteExecutor = new MasterCatalogExecutor();
- try {
- remoteExecutor.forward(id, -1, -1);
- } catch (Exception e) {
- Util.logAndThrowRuntimeException(LOG,
- String.format("failed to forward init catalog %s operation to master.", name), e);
- }
- return;
- }
- init();
- }
+ protected void initLocalObjectsImpl() {
+ esRestClient = new EsRestClient(this.nodes, this.username, this.password, this.enableSsl);
}
- private void init() {
+ @Override
+ protected void init() {
InitCatalogLog initCatalogLog = new InitCatalogLog();
this.esRestClient = new EsRestClient(this.nodes, this.username, this.password, this.enableSsl);
initCatalogLog.setCatalogId(id);
@@ -161,7 +139,6 @@ public class EsExternalCatalog extends ExternalCatalog {
idToDb.put(defaultDbId, db);
initCatalogLog.addCreateDb(defaultDbId, DEFAULT_DB);
}
- initialized = true;
Env.getCurrentEnv().getEditLog().logInitCatalog(initCatalogLog);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index e6e9609ab3..ff82a4dda0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -17,14 +17,17 @@
package org.apache.doris.datasource;
+import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.external.EsExternalDatabase;
import org.apache.doris.catalog.external.ExternalDatabase;
import org.apache.doris.catalog.external.HMSExternalDatabase;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
+import org.apache.doris.common.util.Util;
import org.apache.doris.persist.gson.GsonPostProcessable;
import org.apache.doris.persist.gson.GsonUtils;
+import org.apache.doris.qe.MasterCatalogExecutor;
import com.google.common.collect.Maps;
import com.google.gson.annotations.SerializedName;
@@ -58,14 +61,14 @@ public abstract class ExternalCatalog implements CatalogIf<ExternalDatabase>, Wr
@SerializedName(value = "catalogProperty")
protected CatalogProperty catalogProperty = new CatalogProperty();
@SerializedName(value = "initialized")
- protected boolean initialized = false;
+ private boolean initialized = false;
// Cache of db name to db id
@SerializedName(value = "idToDb")
protected Map<Long, ExternalDatabase> idToDb = Maps.newConcurrentMap();
// db name does not contains "default_cluster"
protected Map<String, Long> dbNameToId = Maps.newConcurrentMap();
- protected boolean objectCreated = false;
+ private boolean objectCreated = false;
/**
* @return names of database in this catalog.
@@ -87,10 +90,45 @@ public abstract class ExternalCatalog implements CatalogIf<ExternalDatabase>, Wr
*/
public abstract boolean tableExist(SessionContext ctx, String dbName, String tblName);
- public abstract void makeSureInitialized();
+ /**
+ * Catalog can't be init when creating because the external catalog may depend on third system.
+ * So you have to make sure the client of third system is initialized before any method was called.
+ */
+ public final synchronized void makeSureInitialized() {
+ initLocalObjects();
+ if (!initialized) {
+ if (!Env.getCurrentEnv().isMaster()) {
+ // Forward to master and wait the journal to replay.
+ MasterCatalogExecutor remoteExecutor = new MasterCatalogExecutor();
+ try {
+ remoteExecutor.forward(id, -1, -1);
+ } catch (Exception e) {
+ Util.logAndThrowRuntimeException(LOG,
+ String.format("failed to forward init catalog %s operation to master.", name), e);
+ }
+ return;
+ }
+ init();
+ initialized = true;
+ }
+ }
+
+ protected final void initLocalObjects() {
+ if (!objectCreated) {
+ initLocalObjectsImpl();
+ objectCreated = true;
+ }
+ }
+
+ // init some local objects such as:
+ // hms client, read properties from hive-site.xml, es client
+ protected abstract void initLocalObjectsImpl();
+
+ // init schema related objects
+ protected abstract void init();
- public void setInitialized(boolean initialized) {
- this.initialized = initialized;
+ public void setUninitialized() {
+ this.initialized = false;
}
public ExternalDatabase getDbForReplay(long dbId) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
index 5896c96622..f348e93787 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
@@ -22,17 +22,12 @@ import org.apache.doris.catalog.HiveMetaStoreClientHelper;
import org.apache.doris.catalog.external.ExternalDatabase;
import org.apache.doris.catalog.external.HMSExternalDatabase;
import org.apache.doris.cluster.ClusterNamespace;
-import org.apache.doris.common.DdlException;
-import org.apache.doris.common.util.Util;
-import org.apache.doris.qe.MasterCatalogExecutor;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import org.apache.hadoop.hive.conf.HiveConf;
-import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import org.apache.thrift.TException;
import org.jetbrains.annotations.Nullable;
import java.util.List;
@@ -44,7 +39,8 @@ import java.util.Map;
public class HMSExternalCatalog extends ExternalCatalog {
private static final Logger LOG = LogManager.getLogger(HMSExternalCatalog.class);
- protected IMetaStoreClient client;
+ private static final int MAX_CLIENT_POOL_SIZE = 8;
+ protected PooledHiveMetaStoreClient client;
/**
* Default constructor for HMSExternalCatalog.
@@ -61,23 +57,15 @@ public class HMSExternalCatalog extends ExternalCatalog {
return catalogProperty.getOrDefault("hive.metastore.uris", "");
}
- private void init() {
+ @Override
+ protected void init() {
Map<String, Long> tmpDbNameToId = Maps.newConcurrentMap();
Map<Long, ExternalDatabase> tmpIdToDb = Maps.newConcurrentMap();
InitCatalogLog initCatalogLog = new InitCatalogLog();
initCatalogLog.setCatalogId(id);
initCatalogLog.setType(InitCatalogLog.Type.HMS);
- List<String> allDatabases;
- try {
- allDatabases = client.getAllDatabases();
- } catch (TException e) {
- LOG.warn("Fail to init db name to id map. {}", e.getMessage());
- return;
- }
+ List<String> allDatabases = client.getAllDatabases();
// Update the db name to id map.
- if (allDatabases == null) {
- return;
- }
for (String dbName : allDatabases) {
long dbId;
if (dbNameToId != null && dbNameToId.containsKey(dbName)) {
@@ -97,41 +85,22 @@ public class HMSExternalCatalog extends ExternalCatalog {
}
dbNameToId = tmpDbNameToId;
idToDb = tmpIdToDb;
- initialized = true;
Env.getCurrentEnv().getEditLog().logInitCatalog(initCatalogLog);
}
- /**
- * Catalog can't be init when creating because the external catalog may depend on third system.
- * So you have to make sure the client of third system is initialized before any method was called.
- */
@Override
- public synchronized void makeSureInitialized() {
- if (!objectCreated) {
- try {
- HiveConf hiveConf = new HiveConf();
- hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, getHiveMetastoreUris());
- client = HiveMetaStoreClientHelper.getClient(hiveConf);
- objectCreated = true;
- } catch (DdlException e) {
- Util.logAndThrowRuntimeException(LOG,
- String.format("failed to create hive meta store client for catalog: %s", name), e);
- }
- }
- if (!initialized) {
- if (!Env.getCurrentEnv().isMaster()) {
- // Forward to master and wait the journal to replay.
- MasterCatalogExecutor remoteExecutor = new MasterCatalogExecutor();
- try {
- remoteExecutor.forward(id, -1, -1);
- } catch (Exception e) {
- Util.logAndThrowRuntimeException(LOG,
- String.format("failed to forward init catalog %s operation to master.", name), e);
- }
- return;
- }
- init();
- }
+ protected void initLocalObjectsImpl() {
+ HiveConf hiveConf = new HiveConf();
+ hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, getHiveMetastoreUris());
+
+ // 1. read properties from hive-site.xml.
+ // and then use properties in CatalogProperty to override properties got from hive-site.xml
+ Map<String, String> properties = HiveMetaStoreClientHelper.getPropertiesForDLF(name, hiveConf);
+ properties.putAll(catalogProperty.getProperties());
+ catalogProperty.setProperties(properties);
+
+ // 2. init hms client
+ client = new PooledHiveMetaStoreClient(hiveConf, MAX_CLIENT_POOL_SIZE);
}
@Override
@@ -149,25 +118,13 @@ public class HMSExternalCatalog extends ExternalCatalog {
hmsExternalDatabase.getTables().stream().forEach(table -> names.add(table.getName()));
return names;
} else {
- try {
- return client.getAllTables(getRealTableName(dbName));
- } catch (TException e) {
- Util.logAndThrowRuntimeException(LOG, String.format("list table names failed for %s.%s", name, dbName),
- e);
- }
+ return client.getAllTables(getRealTableName(dbName));
}
- return Lists.newArrayList();
}
@Override
public boolean tableExist(SessionContext ctx, String dbName, String tblName) {
- try {
- return client.tableExists(getRealTableName(dbName), tblName);
- } catch (TException e) {
- Util.logAndThrowRuntimeException(LOG,
- String.format("check table exist failed for %s.%s.%s", name, dbName, tblName), e);
- }
- return false;
+ return client.tableExists(getRealTableName(dbName), tblName);
}
@Nullable
@@ -198,7 +155,7 @@ public class HMSExternalCatalog extends ExternalCatalog {
return idToDb.get(dbId);
}
- public IMetaStoreClient getClient() {
+ public PooledHiveMetaStoreClient getClient() {
makeSureInitialized();
return client;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index a729f9e71f..c66ee75e46 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -181,7 +181,7 @@ import com.google.common.collect.Sets;
import lombok.Getter;
import org.apache.commons.collections.CollectionUtils;
import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.jetbrains.annotations.Nullable;
@@ -1215,6 +1215,16 @@ public class InternalCatalog implements CatalogIf<Database> {
} else {
typeDef = new TypeDef(resultExpr.getType());
}
+ if (i == 0) {
+ // If this is the first column, because olap table does not support the first column to be
+ // string, float, double or array, we should check and modify its type
+ // For string type, change it to varchar.
+ // For other unsupport types, just remain unchanged, the analysis phash of create table stmt
+ // will handle it.
+ if (typeDef.getType() == Type.STRING) {
+ typeDef = TypeDef.createVarchar(ScalarType.MAX_VARCHAR_LENGTH);
+ }
+ }
ColumnDef columnDef;
if (resultExpr.getSrcSlotRef() == null) {
columnDef = new ColumnDef(name, typeDef, false, null, true, new DefaultValue(false, null), "");
@@ -2268,10 +2278,11 @@ public class InternalCatalog implements CatalogIf<Database> {
HiveTable hiveTable = new HiveTable(tableId, tableName, columns, stmt.getProperties());
hiveTable.setComment(stmt.getComment());
// check hive table whether exists in hive database
- IMetaStoreClient hiveMetaStoreClient = HiveMetaStoreClientHelper.getClient(
- hiveTable.getHiveProperties().get(HiveTable.HIVE_METASTORE_URIS));
- if (!HiveMetaStoreClientHelper.tableExists(hiveMetaStoreClient, hiveTable.getHiveDb(),
- hiveTable.getHiveTable())) {
+ HiveConf hiveConf = new HiveConf();
+ hiveConf.set(HiveMetaStoreClientHelper.HIVE_METASTORE_URIS,
+ hiveTable.getHiveProperties().get(HiveMetaStoreClientHelper.HIVE_METASTORE_URIS));
+ PooledHiveMetaStoreClient client = new PooledHiveMetaStoreClient(hiveConf, 1);
+ if (!client.tableExists(hiveTable.getHiveDb(), hiveTable.getHiveTable())) {
throw new DdlException(String.format("Table [%s] dose not exist in Hive.", hiveTable.getHiveDbTable()));
}
// check hive table if exists in doris database
@@ -2290,15 +2301,17 @@ public class InternalCatalog implements CatalogIf<Database> {
// check hudi properties in create stmt.
HudiUtils.validateCreateTable(hudiTable);
// check hudi table whether exists in hive database
- String metastoreUris = hudiTable.getTableProperties().get(HudiProperty.HUDI_HIVE_METASTORE_URIS);
- IMetaStoreClient hiveMetaStoreClient = HiveMetaStoreClientHelper.getClient(metastoreUris);
- if (!HiveMetaStoreClientHelper.tableExists(hiveMetaStoreClient, hudiTable.getHmsDatabaseName(),
- hudiTable.getHmsTableName())) {
+ HiveConf hiveConf = new HiveConf();
+ hiveConf.set(HiveMetaStoreClientHelper.HIVE_METASTORE_URIS,
+ hudiTable.getTableProperties().get(HudiProperty.HUDI_HIVE_METASTORE_URIS));
+ PooledHiveMetaStoreClient client = new PooledHiveMetaStoreClient(hiveConf, 1);
+ if (!client.tableExists(hudiTable.getHmsDatabaseName(), hudiTable.getHmsTableName())) {
throw new DdlException(
String.format("Table [%s] dose not exist in Hive Metastore.", hudiTable.getHmsTableIdentifer()));
}
- org.apache.hadoop.hive.metastore.api.Table hiveTable = HiveMetaStoreClientHelper.getTable(
- hudiTable.getHmsDatabaseName(), hudiTable.getHmsTableName(), metastoreUris);
+
+ org.apache.hadoop.hive.metastore.api.Table hiveTable = client.getTable(
+ hudiTable.getHmsDatabaseName(), hudiTable.getHmsTableName());
if (!HudiUtils.isHudiTable(hiveTable)) {
throw new DdlException(String.format("Table [%s] is not a hudi table.", hudiTable.getHmsTableIdentifer()));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PooledHiveMetaStoreClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PooledHiveMetaStoreClient.java
new file mode 100644
index 0000000000..c2f3567f56
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PooledHiveMetaStoreClient.java
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource;
+
+import org.apache.doris.catalog.HiveMetaStoreClientHelper;
+import org.apache.doris.common.Config;
+
+import com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient;
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.apache.hadoop.hive.metastore.HiveMetaHookLoader;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Queue;
+
+/**
+ * A hive metastore client pool for a specific hive conf.
+ */
+public class PooledHiveMetaStoreClient {
+ private static final Logger LOG = LogManager.getLogger(PooledHiveMetaStoreClient.class);
+
+ private Queue<CachedClient> clientPool = new LinkedList<>();
+ private static final HiveMetaHookLoader DUMMY_HOOK_LOADER = t -> null;
+ private final int poolSize;
+ private final HiveConf hiveConf;
+
+ public PooledHiveMetaStoreClient(HiveConf hiveConf, int pooSize) {
+ Preconditions.checkArgument(pooSize > 0, pooSize);
+ hiveConf.set(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
+ String.valueOf(Config.hive_metastore_client_timeout_second));
+ this.hiveConf = hiveConf;
+ this.poolSize = pooSize;
+ }
+
+ public List<String> getAllDatabases() {
+ try (CachedClient client = getClient()) {
+ return client.client.getAllDatabases();
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public List<String> getAllTables(String dbName) {
+ try (CachedClient client = getClient()) {
+ return client.client.getAllTables(dbName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public boolean tableExists(String dbName, String tblName) {
+ try (CachedClient client = getClient()) {
+ return client.client.tableExists(dbName, tblName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public boolean listPartitionsByExpr(String dbName, String tblName,
+ byte[] partitionPredicatesInBytes, List<Partition> hivePartitions) {
+ try (CachedClient client = getClient()) {
+ return client.client.listPartitionsByExpr(dbName, tblName, partitionPredicatesInBytes,
+ null, (short) -1, hivePartitions);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public Table getTable(String dbName, String tblName) {
+ try (CachedClient client = getClient()) {
+ return client.client.getTable(dbName, tblName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ public List<FieldSchema> getSchema(String dbName, String tblName) {
+ try (CachedClient client = getClient()) {
+ return client.client.getSchema(dbName, tblName);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private class CachedClient implements AutoCloseable {
+ private final IMetaStoreClient client;
+
+ private CachedClient(HiveConf hiveConf) throws MetaException {
+ String type = hiveConf.get(HiveMetaStoreClientHelper.HIVE_METASTORE_TYPE);
+ if (HiveMetaStoreClientHelper.DLF_TYPE.equalsIgnoreCase(type)) {
+ client = RetryingMetaStoreClient.getProxy(hiveConf, DUMMY_HOOK_LOADER,
+ ProxyMetaStoreClient.class.getName());
+ } else {
+ client = RetryingMetaStoreClient.getProxy(hiveConf, DUMMY_HOOK_LOADER,
+ HiveMetaStoreClient.class.getName());
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ synchronized (clientPool) {
+ if (clientPool.size() > poolSize) {
+ client.close();
+ } else {
+ clientPool.offer(this);
+ }
+ }
+ }
+ }
+
+ private CachedClient getClient() throws MetaException {
+ synchronized (clientPool) {
+ CachedClient client = clientPool.poll();
+ if (client == null) {
+ return new CachedClient(hiveConf);
+ }
+ return client;
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/external/HiveScanProvider.java b/fe/fe-core/src/main/java/org/apache/doris/planner/external/HiveScanProvider.java
index 85425f4cb2..9fd59fc43d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/external/HiveScanProvider.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/external/HiveScanProvider.java
@@ -27,6 +27,7 @@ import org.apache.doris.catalog.HiveMetaStoreClientHelper;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.external.HMSExternalTable;
import org.apache.doris.common.DdlException;
+import org.apache.doris.common.FeConstants;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.UserException;
import org.apache.doris.external.hive.util.HiveUtil;
@@ -65,7 +66,7 @@ public class HiveScanProvider extends HMSTableScanProvider {
private static final Logger LOG = LogManager.getLogger(HiveScanProvider.class);
private static final String PROP_FIELD_DELIMITER = "field.delim";
- private static final String DEFAULT_FIELD_DELIMITER = "\001";
+ private static final String DEFAULT_FIELD_DELIMITER = "\1"; // "\x01"
private static final String DEFAULT_LINE_DELIMITER = "\n";
protected HMSExternalTable hmsTable;
@@ -101,13 +102,21 @@ public class HiveScanProvider extends HMSTableScanProvider {
public TFileType getLocationType() throws DdlException, MetaNotFoundException {
String location = hmsTable.getRemoteTable().getSd().getLocation();
if (location != null && !location.isEmpty()) {
- if (location.startsWith("s3a") || location.startsWith("s3n")) {
+ if (location.startsWith(FeConstants.FS_PREFIX_S3)
+ || location.startsWith(FeConstants.FS_PREFIX_S3A)
+ || location.startsWith(FeConstants.FS_PREFIX_S3N)
+ || location.startsWith(FeConstants.FS_PREFIX_BOS)
+ || location.startsWith(FeConstants.FS_PREFIX_COS)
+ || location.startsWith(FeConstants.FS_PREFIX_OSS)
+ || location.startsWith(FeConstants.FS_PREFIX_OBS)) {
return TFileType.FILE_S3;
- } else if (location.startsWith("hdfs:")) {
+ } else if (location.startsWith(FeConstants.FS_PREFIX_HDFS)) {
return TFileType.FILE_HDFS;
+ } else if (location.startsWith(FeConstants.FS_PREFIX_FILE)) {
+ return TFileType.FILE_LOCAL;
}
}
- throw new DdlException("Unknown file location for hms table.");
+ throw new DdlException("Unknown file location " + location + " for hms table " + hmsTable.getName());
}
@Override
@@ -117,7 +126,10 @@ public class HiveScanProvider extends HMSTableScanProvider {
@Override
public List<InputSplit> getSplits(List<Expr> exprs) throws IOException, UserException {
- String splitsPath = getRemoteHiveTable().getSd().getLocation();
+ // eg:
+ // oss://buckts/data_dir
+ // hdfs://hosts/data_dir
+ String location = getRemoteHiveTable().getSd().getLocation();
List<String> partitionKeys = getRemoteHiveTable().getPartitionKeys().stream().map(FieldSchema::getName)
.collect(Collectors.toList());
List<Partition> hivePartitions = new ArrayList<>();
@@ -146,7 +158,7 @@ public class HiveScanProvider extends HMSTableScanProvider {
throw new IOException(e);
}
} else {
- splits = getSplitsByPath(inputFormat, configuration, splitsPath);
+ splits = getSplitsByPath(inputFormat, configuration, location);
}
return HiveBucketUtil.getPrunedSplitsByBuckets(splits, hmsTable.getName(), exprs,
getRemoteHiveTable().getSd().getBucketCols(), getRemoteHiveTable().getSd().getNumBuckets(),
@@ -154,7 +166,8 @@ public class HiveScanProvider extends HMSTableScanProvider {
}
private List<InputSplit> getSplitsByPath(InputFormat<?, ?> inputFormat, Configuration configuration,
- String splitsPath) throws IOException {
+ String location) throws IOException {
+ String finalLocation = convertToS3IfNecessary(location);
JobConf jobConf = new JobConf(configuration);
// For Tez engine, it may generate subdirectoies for "union" query.
// So there may be files and directories in the table directory at the same time. eg:
@@ -165,16 +178,32 @@ public class HiveScanProvider extends HMSTableScanProvider {
// Otherwise, getSplits() may throw exception: "Not a file xxx"
// https://blog.actorsfit.com/a?ID=00550-ce56ec63-1bff-4b0c-a6f7-447b93efaa31
jobConf.set("mapreduce.input.fileinputformat.input.dir.recursive", "true");
- FileInputFormat.setInputPaths(jobConf, splitsPath);
+ FileInputFormat.setInputPaths(jobConf, finalLocation);
InputSplit[] splits = inputFormat.getSplits(jobConf, 0);
return Lists.newArrayList(splits);
}
+ // convert oss:// to s3://
+ private String convertToS3IfNecessary(String location) throws IOException {
+ LOG.debug("try convert location to s3 prefix: " + location);
+ if (location.startsWith(FeConstants.FS_PREFIX_COS)
+ || location.startsWith(FeConstants.FS_PREFIX_BOS)
+ || location.startsWith(FeConstants.FS_PREFIX_BOS)
+ || location.startsWith(FeConstants.FS_PREFIX_OSS)
+ || location.startsWith(FeConstants.FS_PREFIX_S3A)
+ || location.startsWith(FeConstants.FS_PREFIX_S3N)) {
+ int pos = location.indexOf("://");
+ if (pos == -1) {
+ throw new IOException("No '://' found in location: " + location);
+ }
+ return "s3" + location.substring(pos);
+ }
+ return location;
+ }
protected Configuration setConfiguration() {
Configuration conf = new HdfsConfiguration();
- Map<String, String> dfsProperties = hmsTable.getDfsProperties();
- for (Map.Entry<String, String> entry : dfsProperties.entrySet()) {
+ for (Map.Entry<String, String> entry : hmsTable.getCatalog().getCatalogProperty().getProperties().entrySet()) {
conf.set(entry.getKey(), entry.getValue());
}
Map<String, String> s3Properties = hmsTable.getS3Properties();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/external/QueryScanProvider.java b/fe/fe-core/src/main/java/org/apache/doris/planner/external/QueryScanProvider.java
index 70154ce59c..4b865560d7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/external/QueryScanProvider.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/external/QueryScanProvider.java
@@ -70,6 +70,9 @@ public abstract class QueryScanProvider implements FileScanProviderIf {
String fullPath = ((FileSplit) inputSplits.get(0)).getPath().toUri().toString();
String filePath = ((FileSplit) inputSplits.get(0)).getPath().toUri().getPath();
+ // eg:
+ // hdfs://namenode
+ // s3://buckets
String fsName = fullPath.replace(filePath, "");
TFileType locationType = getLocationType();
context.params.setFileType(locationType);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 14/39: [improvement](profile) support ordinary user to get query profile via http api (#14016)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 76b3c2b4a44950eaa96a1399140d07b7b4757a6c
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Tue Nov 8 20:39:01 2022 +0800
[improvement](profile) support ordinary user to get query profile via http api (#14016)
---
.../fe/manager/query-profile-action.md | 101 ++++++++++++++++++++
.../fe/manager/query-profile-action.md | 102 +++++++++++++++++++++
.../apache/doris/common/util/ProfileManager.java | 25 ++++-
.../httpv2/rest/manager/QueryProfileAction.java | 63 ++++++++++---
4 files changed, 277 insertions(+), 14 deletions(-)
diff --git a/docs/en/docs/admin-manual/http-actions/fe/manager/query-profile-action.md b/docs/en/docs/admin-manual/http-actions/fe/manager/query-profile-action.md
index 0b86949483..efce4a26b8 100644
--- a/docs/en/docs/admin-manual/http-actions/fe/manager/query-profile-action.md
+++ b/docs/en/docs/admin-manual/http-actions/fe/manager/query-profile-action.md
@@ -30,6 +30,8 @@ under the License.
`GET /rest/v2/manager/query/query_info`
+`GET /rest/v2/manager/query/trace/{trace_id}`
+
`GET /rest/v2/manager/query/sql/{query_id}`
`GET /rest/v2/manager/query/profile/text/{query_id}`
@@ -96,6 +98,12 @@ Gets information about select queries for all fe nodes in the cluster.
}
```
+<version since="1.2">
+
+Admin 和 Root 用户可以查看所有 Query。普通用户仅能查看自己发送的 Query。
+
+</version>
+
### Examples
```
GET /rest/v2/manager/query/query_info
@@ -135,6 +143,54 @@ GET /rest/v2/manager/query/query_info
}
```
+## Get Query Id By Trace Id
+
+`GET /rest/v2/manager/query/trace_id/{trace_id}`
+
+### Description
+
+Get query id by trance id.
+
+Before executing a Query, set a unique trace id:
+
+`set set session_context="trace_id:your_trace_id";`
+
+After executing the Query within the same Session, the query id can be obtained through the trace id.
+
+### Path parameters
+
+* `{trace_id}`
+
+ User specific trace id.
+
+### Query parameters
+
+### Response
+
+```
+{
+ "msg": "success",
+ "code": 0,
+ "data": "fb1d9737de914af1-a498d5c5dec638d3",
+ "count": 0
+}
+```
+
+<version since="1.2">
+
+Admin and Root user can view all queries. Ordinary users can only view the Query sent by themselves. If the specified trace id does not exist or has no permission, it will return Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
+
## Get the sql and text profile for the specified query
`GET /rest/v2/manager/query/sql/{query_id}`
@@ -180,6 +236,21 @@ Get the sql and profile text for the specified query id.
"count": 0
}
```
+
+<version since="1.2">
+
+Admin and Root user can view all queries. Ordinary users can only view the Query sent by themselves. If the specified trace id does not exist or has no permission, it will return Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
### Examples
@@ -237,6 +308,21 @@ Get the fragment name, instance id and execution time for the specified query id
"count": 0
}
```
+
+<version since="1.2">
+
+Admin and Root user can view all queries. Ordinary users can only view the Query sent by themselves. If the specified trace id does not exist or has no permission, it will return Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
### Examples
@@ -313,6 +399,21 @@ Get the tree profile information of the specified query id, same as `show query
}
```
+<version since="1.2">
+
+Admin and Root user can view all queries. Ordinary users can only view the Query sent by themselves. If the specified trace id does not exist or has no permission, it will return Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
+
## Current running queries
`GET /rest/v2/manager/query/current_queries`
diff --git a/docs/zh-CN/docs/admin-manual/http-actions/fe/manager/query-profile-action.md b/docs/zh-CN/docs/admin-manual/http-actions/fe/manager/query-profile-action.md
index 5283107279..3954d129cc 100644
--- a/docs/zh-CN/docs/admin-manual/http-actions/fe/manager/query-profile-action.md
+++ b/docs/zh-CN/docs/admin-manual/http-actions/fe/manager/query-profile-action.md
@@ -30,6 +30,8 @@ under the License.
`GET /rest/v2/manager/query/query_info`
+`GET /rest/v2/manager/query/trace/{trace_id}`
+
`GET /rest/v2/manager/query/sql/{query_id}`
`GET /rest/v2/manager/query/profile/text/{query_id}`
@@ -96,6 +98,12 @@ under the License.
}
```
+<version since="1.2">
+
+Admin 和 Root 用户可以查看所有 Query。普通用户仅能查看自己发送的 Query。
+
+</version>
+
### Examples
```
GET /rest/v2/manager/query/query_info
@@ -135,6 +143,54 @@ GET /rest/v2/manager/query/query_info
}
```
+## 通过 Trace Id 获取 Query Id
+
+`GET /rest/v2/manager/query/trace_id/{trace_id}`
+
+### Description
+
+通过 Trace Id 获取 Query Id.
+
+在执行一个 Query 前,先设置一个唯一的 trace id:
+
+`set set session_context="trace_id:your_trace_id";`
+
+在同一个 Session 链接内执行 Query 后,可以通过 trace id 获取 query id。
+
+### Path parameters
+
+* `{trace_id}`
+
+ 用户设置的 trace id.
+
+### Query parameters
+
+### Response
+
+```
+{
+ "msg": "success",
+ "code": 0,
+ "data": "fb1d9737de914af1-a498d5c5dec638d3",
+ "count": 0
+}
+```
+
+<version since="1.2">
+
+Admin 和 Root 用户可以查看所有 Query。普通用户仅能查看自己发送的 Query。若指定 trace id 不存在或无权限,则返回 Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
+
## 获取指定查询的sql和文本profile
`GET /rest/v2/manager/query/sql/{query_id}`
@@ -180,6 +236,21 @@ GET /rest/v2/manager/query/query_info
"count": 0
}
```
+
+<version since="1.2">
+
+Admin 和 Root 用户可以查看所有 Query。普通用户仅能查看自己发送的 Query。若指定 query id 不存在或无权限,则返回 Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
### Examples
@@ -237,6 +308,21 @@ GET /rest/v2/manager/query/query_info
"count": 0
}
```
+
+<version since="1.2">
+
+Admin 和 Root 用户可以查看所有 Query。普通用户仅能查看自己发送的 Query。若指定 query id 不存在或无权限,则返回 Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
### Examples
@@ -313,6 +399,21 @@ GET /rest/v2/manager/query/query_info
}
```
+<version since="1.2">
+
+Admin 和 Root 用户可以查看所有 Query。普通用户仅能查看自己发送的 Query。若指定 query id 不存在或无权限,则返回 Bad Request:
+
+```
+{
+ "msg": "Bad Request",
+ "code": 403,
+ "data": "error messages",
+ "count": 0
+}
+```
+
+</version>
+
## 正在执行的query
`GET /rest/v2/manager/query/current_queries`
@@ -372,3 +473,4 @@ GET /rest/v2/manager/query/query_info
"count": 0
}
```
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/ProfileManager.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/ProfileManager.java
index 1dc0125707..3922c496c7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/ProfileManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/ProfileManager.java
@@ -18,7 +18,9 @@
package org.apache.doris.common.util;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.AuthenticationException;
import org.apache.doris.common.Config;
+import org.apache.doris.common.DdlException;
import org.apache.doris.common.profile.MultiProfileTreeBuilder;
import org.apache.doris.common.profile.ProfileTreeBuilder;
import org.apache.doris.common.profile.ProfileTreeNode;
@@ -208,13 +210,34 @@ public class ProfileManager {
if (element == null) {
return null;
}
-
return element.profileContent;
} finally {
readLock.unlock();
}
}
+ /**
+ * Check if the query with specific query id is queried by specific user.
+ *
+ * @param user
+ * @param queryId
+ * @throws DdlException
+ */
+ public void checkAuthByUserAndQueryId(String user, String queryId) throws AuthenticationException {
+ readLock.lock();
+ try {
+ ProfileElement element = queryIdToProfileMap.get(queryId);
+ if (element == null) {
+ throw new AuthenticationException("query with id " + queryId + " not found");
+ }
+ if (!element.infoStrings.get(USER).equals(user)) {
+ throw new AuthenticationException("Access deny to view query with id: " + queryId);
+ }
+ } finally {
+ readLock.unlock();
+ }
+ }
+
public ProfileTreeNode getFragmentProfileTree(String queryID, String executionId) throws AnalysisException {
MultiProfileTreeBuilder builder;
readLock.lock();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java
index b395990f74..f4cbcf76f9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java
@@ -19,6 +19,7 @@ package org.apache.doris.httpv2.rest.manager;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.AuthenticationException;
import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
import org.apache.doris.common.proc.CurrentQueryStatementsProcNode;
@@ -28,6 +29,7 @@ import org.apache.doris.common.profile.ProfileTreePrinter;
import org.apache.doris.common.util.ProfileManager;
import org.apache.doris.httpv2.entity.ResponseEntityBuilder;
import org.apache.doris.httpv2.rest.RestBaseController;
+import org.apache.doris.mysql.privilege.PaloAuth;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.qe.ConnectContext;
@@ -59,6 +61,7 @@ import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
+import java.util.stream.Stream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@@ -133,7 +136,6 @@ public class QueryProfileAction extends RestBaseController {
@RequestParam(value = IS_ALL_NODE_PARA, required = false, defaultValue = "true")
boolean isAllNode) {
executeCheckPassword(request, response);
- checkGlobalAuth(ConnectContext.get().getCurrentUserIdentity(), PrivPredicate.ADMIN);
List<List<String>> queries = Lists.newArrayList();
if (isAllNode) {
@@ -166,11 +168,10 @@ public class QueryProfileAction extends RestBaseController {
return ResponseEntityBuilder.ok(new NodeAction.NodeInfo(QUERY_TITLE_NAMES, queries));
}
- queries = ProfileManager.getInstance().getAllQueries().stream()
- .filter(profile -> profile.get(4).equals("Query")).collect(Collectors.toList());
- if (!Strings.isNullOrEmpty(queryId)) {
- queries = queries.stream().filter(q -> q.get(0).equals(queryId)).collect(Collectors.toList());
- }
+ Stream<List<String>> queryStream = ProfileManager.getInstance().getAllQueries().stream()
+ .filter(profile -> profile.get(4).equals("Query"));
+ queryStream = filterQueriesByUserAndQueryId(queryStream, queryId);
+ queries = queryStream.collect(Collectors.toList());
// add node information
for (List<String> query : queries) {
@@ -200,7 +201,6 @@ public class QueryProfileAction extends RestBaseController {
@RequestParam(value = IS_ALL_NODE_PARA, required = false, defaultValue = "true")
boolean isAllNode) {
executeCheckPassword(request, response);
- checkGlobalAuth(ConnectContext.get().getCurrentUserIdentity(), PrivPredicate.ADMIN);
Map<String, String> querySql = Maps.newHashMap();
if (isAllNode) {
@@ -219,9 +219,9 @@ public class QueryProfileAction extends RestBaseController {
}
}
} else {
- List<List<String>> queries =
- ProfileManager.getInstance().getAllQueries().stream().filter(query -> query.get(0).equals(queryId))
- .collect(Collectors.toList());
+ Stream<List<String>> queryStream = ProfileManager.getInstance().getAllQueries().stream();
+ queryStream = filterQueriesByUserAndQueryId(queryStream, queryId);
+ List<List<String>> queries = queryStream.collect(Collectors.toList());
if (!queries.isEmpty()) {
querySql.put("sql", queries.get(0).get(3));
}
@@ -229,6 +229,20 @@ public class QueryProfileAction extends RestBaseController {
return ResponseEntityBuilder.ok(querySql);
}
+ private Stream<List<String>> filterQueriesByUserAndQueryId(Stream<List<String>> queryStream, String queryId) {
+ // filter by user
+ // Only admin or root user can see all profile.
+ // Common user can only review the query of their own.
+ String user = ConnectContext.get().getCurrentUserIdentity().getQualifiedUser();
+ if (!user.equalsIgnoreCase(PaloAuth.ADMIN_USER) && !user.equalsIgnoreCase(PaloAuth.ROOT_USER)) {
+ queryStream = queryStream.filter(q -> q.get(1).equals(user));
+ }
+ if (!Strings.isNullOrEmpty(queryId)) {
+ queryStream = queryStream.filter(query -> query.get(0).equals(queryId));
+ }
+ return queryStream;
+ }
+
/**
* Returns the text profile for the specified query id.
* There are 3 formats:
@@ -251,7 +265,12 @@ public class QueryProfileAction extends RestBaseController {
@RequestParam(value = INSTANCE_ID, required = false) String instanceId,
@RequestParam(value = IS_ALL_NODE_PARA, required = false, defaultValue = "true") boolean isAllNode) {
executeCheckPassword(request, response);
- checkGlobalAuth(ConnectContext.get().getCurrentUserIdentity(), PrivPredicate.ADMIN);
+
+ try {
+ checkAuthByUserAndQueryId(queryId);
+ } catch (AuthenticationException e) {
+ return ResponseEntityBuilder.badRequest(e.getMessage());
+ }
if (format.equals("text")) {
return getTextProfile(request, queryId, isAllNode);
@@ -278,7 +297,6 @@ public class QueryProfileAction extends RestBaseController {
@PathVariable("trace_id") String traceId,
@RequestParam(value = IS_ALL_NODE_PARA, required = false, defaultValue = "true") boolean isAllNode) {
executeCheckPassword(request, response);
- checkGlobalAuth(ConnectContext.get().getCurrentUserIdentity(), PrivPredicate.ADMIN);
if (isAllNode) {
String httpPath = "/rest/v2/manager/query/trace_id/" + traceId;
@@ -305,6 +323,13 @@ public class QueryProfileAction extends RestBaseController {
if (Strings.isNullOrEmpty(queryId)) {
return ResponseEntityBuilder.badRequest("Not found");
}
+
+ try {
+ checkAuthByUserAndQueryId(queryId);
+ } catch (AuthenticationException e) {
+ return ResponseEntityBuilder.badRequest(e.getMessage());
+ }
+
return ResponseEntityBuilder.ok(queryId);
}
return ResponseEntityBuilder.badRequest("not found query id");
@@ -315,7 +340,6 @@ public class QueryProfileAction extends RestBaseController {
@PathVariable("query_id") String queryId,
@RequestParam(value = IS_ALL_NODE_PARA, required = false, defaultValue = "true") boolean isAllNode) {
executeCheckPassword(request, response);
- checkGlobalAuth(ConnectContext.get().getCurrentUserIdentity(), PrivPredicate.ADMIN);
if (isAllNode) {
String httpPath = "/rest/v2/manager/query/profile/fragments/" + queryId;
@@ -337,6 +361,12 @@ public class QueryProfileAction extends RestBaseController {
}
}
} else {
+ try {
+ checkAuthByUserAndQueryId(queryId);
+ } catch (AuthenticationException e) {
+ return ResponseEntityBuilder.badRequest(e.getMessage());
+ }
+
try {
return ResponseEntityBuilder.ok(ProfileManager.getInstance().getFragmentsAndInstances(queryId));
} catch (AnalysisException e) {
@@ -439,6 +469,13 @@ public class QueryProfileAction extends RestBaseController {
return ResponseEntityBuilder.ok(result);
}
+ private void checkAuthByUserAndQueryId(String queryId) throws AuthenticationException {
+ String user = ConnectContext.get().getCurrentUserIdentity().getQualifiedUser();
+ if (!user.equalsIgnoreCase(PaloAuth.ADMIN_USER) && !user.equalsIgnoreCase(PaloAuth.ROOT_USER)) {
+ ProfileManager.getInstance().checkAuthByUserAndQueryId(user, queryId);
+ }
+ }
+
/**
* return the result of CurrentQueryStatementsProcNode.
*
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/39: [fix](priv) fix meta replay bug when upgrading from 1.1.x to 1.2.x (#14046)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ab9aef389ff3de4f3fb42bc135b2cc162f98b75c
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Tue Nov 8 10:43:33 2022 +0800
[fix](priv) fix meta replay bug when upgrading from 1.1.x to 1.2.x (#14046)
---
.../java/org/apache/doris/mysql/privilege/PrivBitSet.java | 2 +-
.../org/apache/doris/mysql/privilege/UserPrivTable.java | 2 +-
.../org/apache/doris/mysql/privilege/PrivEntryTest.java | 15 +++++++++++++++
3 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivBitSet.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivBitSet.java
index 3f1b844775..a5a46cb294 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivBitSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PrivBitSet.java
@@ -53,7 +53,7 @@ public class PrivBitSet implements Writable {
public void unset(int index) {
Preconditions.checkState(index < PaloPrivilege.privileges.length, index);
- set ^= 1 << index;
+ set &= ~(1 << index);
}
public boolean get(int index) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPrivTable.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPrivTable.java
index 767ee15b8d..b69b1157d5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPrivTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserPrivTable.java
@@ -209,7 +209,7 @@ public class UserPrivTable extends PrivTable {
try {
// USAGE_PRIV is no need to degrade.
PrivBitSet removeUsagePriv = globalPrivEntry.privSet.copy();
- removeUsagePriv.xor(PrivBitSet.of(PaloPrivilege.USAGE_PRIV));
+ removeUsagePriv.unset(PaloPrivilege.USAGE_PRIV.getIdx());
CatalogPrivEntry entry = CatalogPrivEntry.create(globalPrivEntry.origUser, globalPrivEntry.origHost,
InternalCatalog.INTERNAL_CATALOG_NAME, globalPrivEntry.isDomain, removeUsagePriv);
entry.setSetByDomainResolver(false);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/PrivEntryTest.java b/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/PrivEntryTest.java
index 0a261c0951..16e31f4a68 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/PrivEntryTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/PrivEntryTest.java
@@ -45,4 +45,19 @@ public class PrivEntryTest {
tablePrivTable.getPrivs(userIdentity, InternalCatalog.INTERNAL_CATALOG_NAME, "db_db1", "tbl_tbl1", privs2);
Assert.assertTrue(PaloPrivilege.satisfy(privs2, PrivPredicate.DROP));
}
+
+ @Test
+ public void testPrivBitSet() {
+ PrivBitSet privBitSet = PrivBitSet.of(PaloPrivilege.ADMIN_PRIV, PaloPrivilege.NODE_PRIV);
+ Assert.assertTrue(privBitSet.containsPrivs(PaloPrivilege.ADMIN_PRIV));
+ Assert.assertTrue(privBitSet.containsPrivs(PaloPrivilege.NODE_PRIV));
+ privBitSet.set(PaloPrivilege.DROP_PRIV.getIdx());
+ Assert.assertTrue(privBitSet.containsPrivs(PaloPrivilege.DROP_PRIV));
+ privBitSet.set(PaloPrivilege.DROP_PRIV.getIdx());
+ Assert.assertTrue(privBitSet.containsPrivs(PaloPrivilege.DROP_PRIV));
+ privBitSet.unset(PaloPrivilege.NODE_PRIV.getIdx());
+ Assert.assertFalse(privBitSet.containsPrivs(PaloPrivilege.NODE_PRIV));
+ privBitSet.unset(PaloPrivilege.NODE_PRIV.getIdx());
+ Assert.assertFalse(privBitSet.containsPrivs(PaloPrivilege.NODE_PRIV));
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 35/39: [fix](compile) fix compile error #14103
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 12ca106caef6f06c0695595ee9a6ec2620a09d8b
Author: camby <10...@qq.com>
AuthorDate: Wed Nov 9 14:10:06 2022 +0800
[fix](compile) fix compile error #14103
Co-authored-by: cambyzju <zh...@baidu.com>
---
be/src/vec/exec/vmysql_scan_node.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/be/src/vec/exec/vmysql_scan_node.h b/be/src/vec/exec/vmysql_scan_node.h
index b3adf90b0d..aa364de900 100644
--- a/be/src/vec/exec/vmysql_scan_node.h
+++ b/be/src/vec/exec/vmysql_scan_node.h
@@ -21,6 +21,7 @@
#include "exec/mysql_scanner.h"
#include "exec/scan_node.h"
+#include "exec/text_converter.h"
#include "runtime/descriptors.h"
namespace doris {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 30/39: [docs](array-type) update the docs to specify how to use array function when import data (#13995)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit fdb42a4dd2836206052b23bc5c6ba5c11004cf0e
Author: carlvinhust2012 <hu...@126.com>
AuthorDate: Wed Nov 9 12:21:26 2022 +0800
[docs](array-type) update the docs to specify how to use array function when import data (#13995)
Co-authored-by: hucheng01 <hu...@baidu.com>
---
docs/en/docs/data-operate/import/load-manual.md | 17 +++++++++++++++++
docs/zh-CN/docs/data-operate/import/load-manual.md | 18 ++++++++++++++++++
2 files changed, 35 insertions(+)
diff --git a/docs/en/docs/data-operate/import/load-manual.md b/docs/en/docs/data-operate/import/load-manual.md
index 5bc11a82da..84469e7e05 100644
--- a/docs/en/docs/data-operate/import/load-manual.md
+++ b/docs/en/docs/data-operate/import/load-manual.md
@@ -81,3 +81,20 @@ For best practices on atomicity guarantees, see Importing Transactions and Atomi
## Synchronous and asynchronous imports
Import methods are divided into synchronous and asynchronous. For the synchronous import method, the returned result indicates whether the import succeeds or fails. For the asynchronous import method, a successful return only means that the job was submitted successfully, not that the data was imported successfully. You need to use the corresponding command to check the running status of the import job.
+
+## Import the data of array type
+
+The array function can only be supported in vectorization scenarios, but non-vectorization scenarios are not supported.
+if you want to apply the array function to import data, you should enable vectorization engine. Then you need to cast the input parameter column into the array type according to the parameter of the array function. Finally, you can continue to use the array function.
+
+For example, in the following import, you need to cast columns b14 and a13 into `array<string>` type, and then use the `array_union` function.
+
+```sql
+LOAD LABEL label_03_14_49_34_898986_19090452100 (
+ DATA INFILE("hdfs://test.hdfs.com:9000/user/test/data/sys/load/array_test.data")
+ INTO TABLE `test_array_table`
+ COLUMNS TERMINATED BY "|" (`k1`, `a1`, `a2`, `a3`, `a4`, `a5`, `a6`, `a7`, `a8`, `a9`, `a10`, `a11`, `a12`, `a13`, `b14`)
+ SET(a14=array_union(cast(b14 as array<string>), cast(a13 as array<string>))) WHERE size(a2) > 270)
+ WITH BROKER "hdfs" ("username"="test_array", "password"="")
+ PROPERTIES( "max_filter_ratio"="0.8" );
+```
diff --git a/docs/zh-CN/docs/data-operate/import/load-manual.md b/docs/zh-CN/docs/data-operate/import/load-manual.md
index bdbd8765e7..4d8d80baca 100644
--- a/docs/zh-CN/docs/data-operate/import/load-manual.md
+++ b/docs/zh-CN/docs/data-operate/import/load-manual.md
@@ -82,3 +82,21 @@ Label 是用于保证对应的导入作业,仅能成功导入一次。一个
导入方式分为同步和异步。对于同步导入方式,返回结果即表示导入成功还是失败。而对于异步导入方式,返回成功仅代表作业提交成功,不代表数据导入成功,需要使用对应的命令查看导入作业的运行状态。
+## 导入array类型
+
+向量化场景才能支持array函数,非向量化场景不支持。
+
+如果想要应用array函数导入数据,则应先启用向量化功能;然后需要根据array函数的参数类型将输入参数列转换为array类型;最后,就可以继续使用array函数了。
+
+例如以下导入,需要先将列b14和列a13先cast成`array<string>`类型,再运用`array_union`函数。
+
+```sql
+LOAD LABEL label_03_14_49_34_898986_19090452100 (
+ DATA INFILE("hdfs://test.hdfs.com:9000/user/test/data/sys/load/array_test.data")
+ INTO TABLE `test_array_table`
+ COLUMNS TERMINATED BY "|" (`k1`, `a1`, `a2`, `a3`, `a4`, `a5`, `a6`, `a7`, `a8`, `a9`, `a10`, `a11`, `a12`, `a13`, `b14`)
+ SET(a14=array_union(cast(b14 as array<string>), cast(a13 as array<string>))) WHERE size(a2) > 270)
+ WITH BROKER "hdfs" ("username"="test_array", "password"="")
+ PROPERTIES( "max_filter_ratio"="0.8" );
+```
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 23/39: [enhancement](load) shrink reserved buffer for page builder (#14012) (#14014)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 83f243acf0ddcfd49b81b661889d59268e2994f2
Author: zhengyu <fr...@gmail.com>
AuthorDate: Wed Nov 9 08:40:07 2022 +0800
[enhancement](load) shrink reserved buffer for page builder (#14012) (#14014)
* [enhancement](load) shrink reserved buffer for page builder (#14012)
For table with hundreds of text type columns, flushing its memtable may cost huge memory.
These memory are consumed when initializing page builder, as it reserves 1MB for each column.
So memory consumption grows in proportion with column number. Shrinking the reservation may
reduce memory substantially in load process.
Signed-off-by: freemandealer <fr...@gmail.com>
* response to the review
Signed-off-by: freemandealer <fr...@gmail.com>
* Update binary_plain_page.h
* Update binary_dict_page.cpp
* Update binary_plain_page.h
Signed-off-by: freemandealer <fr...@gmail.com>
---
be/src/olap/rowset/segment_v2/binary_dict_page.cpp | 8 +++++++-
be/src/olap/rowset/segment_v2/binary_plain_page.h | 14 +++++++++++---
be/src/olap/rowset/segment_v2/options.h | 2 ++
3 files changed, 20 insertions(+), 4 deletions(-)
diff --git a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
index b8d3fa98c6..f86c16f134 100644
--- a/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
+++ b/be/src/olap/rowset/segment_v2/binary_dict_page.cpp
@@ -37,7 +37,9 @@ BinaryDictPageBuilder::BinaryDictPageBuilder(const PageBuilderOptions& options)
// TODO: the data page builder type can be created by Factory according to user config
_data_page_builder.reset(new BitshufflePageBuilder<OLAP_FIELD_TYPE_INT>(options));
PageBuilderOptions dict_builder_options;
- dict_builder_options.data_page_size = _options.dict_page_size;
+ dict_builder_options.data_page_size =
+ std::min(_options.data_page_size, _options.dict_page_size);
+ dict_builder_options.is_dict_page = true;
_dict_builder.reset(new BinaryPlainPageBuilder<OLAP_FIELD_TYPE_VARCHAR>(dict_builder_options));
reset();
}
@@ -118,6 +120,10 @@ Status BinaryDictPageBuilder::add(const uint8_t* vals, size_t* count) {
}
OwnedSlice BinaryDictPageBuilder::finish() {
+ if (VLOG_DEBUG_IS_ON && _encoding_type == DICT_ENCODING) {
+ VLOG_DEBUG << "dict page size:" << _dict_builder->size();
+ }
+
DCHECK(!_finished);
_finished = true;
diff --git a/be/src/olap/rowset/segment_v2/binary_plain_page.h b/be/src/olap/rowset/segment_v2/binary_plain_page.h
index 96cfc392a5..9faaeab3eb 100644
--- a/be/src/olap/rowset/segment_v2/binary_plain_page.h
+++ b/be/src/olap/rowset/segment_v2/binary_plain_page.h
@@ -53,8 +53,14 @@ public:
}
bool is_page_full() override {
- // data_page_size is 0, do not limit the page size
- return _options.data_page_size != 0 && _size_estimate > _options.data_page_size;
+ bool ret = false;
+ if (_options.is_dict_page) {
+ // dict_page_size is 0, do not limit the page size
+ ret = _options.dict_page_size != 0 && _size_estimate > _options.dict_page_size;
+ } else {
+ ret = _options.data_page_size != 0 && _size_estimate > _options.data_page_size;
+ }
+ return ret;
}
Status add(const uint8_t* vals, size_t* count) override {
@@ -104,7 +110,9 @@ public:
void reset() override {
_offsets.clear();
_buffer.clear();
- _buffer.reserve(_options.data_page_size == 0 ? 1024 : _options.data_page_size);
+ _buffer.reserve(_options.data_page_size == 0
+ ? 1024
+ : std::min(_options.data_page_size, _options.dict_page_size));
_size_estimate = sizeof(uint32_t);
_finished = false;
_last_value_size = 0;
diff --git a/be/src/olap/rowset/segment_v2/options.h b/be/src/olap/rowset/segment_v2/options.h
index 9405eb19cf..19041f4c51 100644
--- a/be/src/olap/rowset/segment_v2/options.h
+++ b/be/src/olap/rowset/segment_v2/options.h
@@ -30,6 +30,8 @@ struct PageBuilderOptions {
size_t dict_page_size = DEFAULT_PAGE_SIZE;
bool need_check_bitmap = true;
+
+ bool is_dict_page = false; // page used for saving dictionary
};
struct PageDecoderOptions {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 15/39: [Nereids][Improve] infer predicate after push down predicate (#12996)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0e39243080d7464b83ff64f567d081d25ff8a75d
Author: shee <13...@users.noreply.github.com>
AuthorDate: Tue Nov 8 21:36:17 2022 +0800
[Nereids][Improve] infer predicate after push down predicate (#12996)
This PR implements the function of predicate inference
For example:
``` sql
select * from student left join score on student.id = score.sid where score.sid > 1
```
transformed logical plan tree:
left join
/ \
filter(sid >1) filter(id > 1) <---- inferred predicate
| |
scan scan
See `InferPredicatesTest` for more cases
The logic is as follows:
1. poll up bottom predicate then infer additional predicates
for example:
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id
1. poll up bottom predicate
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t.id = 1
2. infer
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t.id = 1 and t2.id = 1
finally transformed sql:
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t2.id = 1
2. put these predicates into `otherJoinConjuncts` , these predicates are processed in the next
round of predicate push-down
Now only support infer `ComparisonPredicate`.
TODO: We should determine whether `expression` satisfies the condition for replacement
eg: Satisfy `expression` is non-deterministic
---
.../org/apache/doris/nereids/jobs/JobType.java | 1 +
.../doris/nereids/jobs/batch/BatchRulesJob.java | 7 +
.../jobs/batch/NereidsRewriteJobExecutor.java | 3 +
.../nereids/jobs/rewrite/VisitorRewriteJob.java | 56 +++
.../rules/rewrite/logical/InferPredicates.java | 117 +++++
.../rewrite/logical/PredicatePropagation.java | 105 ++++
.../rules/rewrite/logical/PullUpPredicates.java | 165 +++++++
.../nereids/trees/expressions/Expression.java | 4 +
.../nereids/trees/plans/logical/LogicalJoin.java | 6 +
.../rules/rewrite/logical/InferPredicatesTest.java | 531 +++++++++++++++++++++
10 files changed, 995 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/JobType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/JobType.java
index 6531376177..528109babb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/JobType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/JobType.java
@@ -29,6 +29,7 @@ public enum JobType {
APPLY_RULE,
DERIVE_STATS,
TOP_DOWN_REWRITE,
+ VISITOR_REWRITE,
BOTTOM_UP_REWRITE
;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/BatchRulesJob.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/BatchRulesJob.java
index c77c0ec8e2..52014728ca 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/BatchRulesJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/BatchRulesJob.java
@@ -19,11 +19,14 @@ package org.apache.doris.nereids.jobs.batch;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.jobs.Job;
+import org.apache.doris.nereids.jobs.JobContext;
import org.apache.doris.nereids.jobs.cascades.OptimizeGroupJob;
import org.apache.doris.nereids.jobs.rewrite.RewriteBottomUpJob;
import org.apache.doris.nereids.jobs.rewrite.RewriteTopDownJob;
+import org.apache.doris.nereids.jobs.rewrite.VisitorRewriteJob;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleFactory;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
import java.util.ArrayList;
import java.util.List;
@@ -71,6 +74,10 @@ public abstract class BatchRulesJob {
cascadesContext.getCurrentJobContext(), once);
}
+ protected Job visitorJob(DefaultPlanRewriter<JobContext> planRewriter) {
+ return new VisitorRewriteJob(cascadesContext, planRewriter, true);
+ }
+
protected Job optimize() {
return new OptimizeGroupJob(
cascadesContext.getMemo().getRoot(),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/NereidsRewriteJobExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/NereidsRewriteJobExecutor.java
index 52311db55a..634e4ca9fb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/NereidsRewriteJobExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/NereidsRewriteJobExecutor.java
@@ -30,6 +30,7 @@ import org.apache.doris.nereids.rules.rewrite.logical.EliminateLimit;
import org.apache.doris.nereids.rules.rewrite.logical.EliminateUnnecessaryProject;
import org.apache.doris.nereids.rules.rewrite.logical.ExtractSingleTableExpressionFromDisjunction;
import org.apache.doris.nereids.rules.rewrite.logical.FindHashConditionForJoin;
+import org.apache.doris.nereids.rules.rewrite.logical.InferPredicates;
import org.apache.doris.nereids.rules.rewrite.logical.LimitPushDown;
import org.apache.doris.nereids.rules.rewrite.logical.NormalizeAggregate;
import org.apache.doris.nereids.rules.rewrite.logical.PruneOlapScanPartition;
@@ -65,9 +66,11 @@ public class NereidsRewriteJobExecutor extends BatchRulesJob {
.add(topDownBatch(ImmutableList.of(new ExtractSingleTableExpressionFromDisjunction())))
.add(topDownBatch(ImmutableList.of(new NormalizeAggregate())))
.add(topDownBatch(RuleSet.PUSH_DOWN_FILTERS, false))
+ .add(visitorJob(new InferPredicates()))
.add(topDownBatch(ImmutableList.of(new ReorderJoin())))
.add(topDownBatch(ImmutableList.of(new ColumnPruning())))
.add(topDownBatch(RuleSet.PUSH_DOWN_FILTERS, false))
+ .add(visitorJob(new InferPredicates()))
.add(topDownBatch(ImmutableList.of(PushFilterInsideJoin.INSTANCE)))
.add(topDownBatch(ImmutableList.of(new FindHashConditionForJoin())))
.add(topDownBatch(ImmutableList.of(new LimitPushDown())))
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/VisitorRewriteJob.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/VisitorRewriteJob.java
new file mode 100644
index 0000000000..f579de23ad
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/rewrite/VisitorRewriteJob.java
@@ -0,0 +1,56 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.jobs.rewrite;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.jobs.Job;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.jobs.JobType;
+import org.apache.doris.nereids.memo.Group;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+
+import java.util.Objects;
+
+/**
+ * Use visitor to rewrite the plan.
+ */
+public class VisitorRewriteJob extends Job {
+ private final Group group;
+
+ private final DefaultPlanRewriter<JobContext> planRewriter;
+
+ /**
+ * Constructor.
+ */
+ public VisitorRewriteJob(CascadesContext cascadesContext, DefaultPlanRewriter<JobContext> rewriter, boolean once) {
+ super(JobType.VISITOR_REWRITE, cascadesContext.getCurrentJobContext(), once);
+ this.group = Objects.requireNonNull(cascadesContext.getMemo().getRoot(), "group cannot be null");
+ this.planRewriter = Objects.requireNonNull(rewriter, "planRewriter cannot be null");
+ }
+
+ @Override
+ public void execute() {
+ GroupExpression logicalExpression = group.getLogicalExpression();
+ Plan root = context.getCascadesContext().getMemo().copyOut(logicalExpression, true);
+ Plan rewrittenRoot = root.accept(planRewriter, context);
+ context.getCascadesContext().getMemo().copyIn(rewrittenRoot, group, true);
+ }
+
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java
new file mode 100644
index 0000000000..d1e5ab77ef
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicates.java
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * infer additional predicates for `LogicalFilter` and `LogicalJoin`.
+ * The logic is as follows:
+ * 1. poll up bottom predicate then infer additional predicates
+ * for example:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id
+ * 1. poll up bottom predicate
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t.id = 1
+ * 2. infer
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t.id = 1 and t2.id = 1
+ * finally transformed sql:
+ * select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t2.id = 1
+ * 2. put these predicates into `otherJoinConjuncts` , these predicates are processed in the next
+ * round of predicate push-down
+ */
+public class InferPredicates extends DefaultPlanRewriter<JobContext> {
+ private final PredicatePropagation propagation = new PredicatePropagation();
+ private final PullUpPredicates pollUpPredicates = new PullUpPredicates();
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, JobContext context) {
+ join = (LogicalJoin<? extends Plan, ? extends Plan>) super.visit(join, context);
+ Plan left = join.left();
+ Plan right = join.right();
+ Set<Expression> expressions = getAllExpressions(left, right, join.getOnClauseCondition());
+ List<Expression> otherJoinConjuncts = Lists.newArrayList(join.getOtherJoinConjuncts());
+ switch (join.getJoinType()) {
+ case INNER_JOIN:
+ case CROSS_JOIN:
+ case LEFT_SEMI_JOIN:
+ case RIGHT_SEMI_JOIN:
+ otherJoinConjuncts.addAll(inferNewPredicate(left, expressions));
+ otherJoinConjuncts.addAll(inferNewPredicate(right, expressions));
+ break;
+ case LEFT_OUTER_JOIN:
+ case LEFT_ANTI_JOIN:
+ otherJoinConjuncts.addAll(inferNewPredicate(right, expressions));
+ break;
+ case RIGHT_OUTER_JOIN:
+ case RIGHT_ANTI_JOIN:
+ otherJoinConjuncts.addAll(inferNewPredicate(left, expressions));
+ break;
+ default:
+ return join;
+ }
+ return join.withOtherJoinConjuncts(otherJoinConjuncts);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, JobContext context) {
+ filter = (LogicalFilter<? extends Plan>) super.visit(filter, context);
+ Set<Expression> filterPredicates = pullUpPredicates(filter);
+ filterPredicates.removeAll(pullUpPredicates(filter.child()));
+ filter.getConjuncts().forEach(filterPredicates::remove);
+ if (!filterPredicates.isEmpty()) {
+ filterPredicates.addAll(filter.getConjuncts());
+ return new LogicalFilter<>(ExpressionUtils.and(Lists.newArrayList(filterPredicates)), filter.child());
+ }
+ return filter;
+ }
+
+ private Set<Expression> getAllExpressions(Plan left, Plan right, Optional<Expression> condition) {
+ Set<Expression> baseExpressions = pullUpPredicates(left);
+ baseExpressions.addAll(pullUpPredicates(right));
+ condition.ifPresent(on -> baseExpressions.addAll(ExpressionUtils.extractConjunction(on)));
+ baseExpressions.addAll(propagation.infer(baseExpressions));
+ return baseExpressions;
+ }
+
+ private Set<Expression> pullUpPredicates(Plan plan) {
+ return Sets.newHashSet(plan.accept(pollUpPredicates, null));
+ }
+
+ private List<Expression> inferNewPredicate(Plan plan, Set<Expression> expressions) {
+ List<Expression> predicates = expressions.stream()
+ .filter(c -> !c.getInputSlots().isEmpty() && plan.getOutputSet().containsAll(
+ c.getInputSlots())).collect(Collectors.toList());
+ predicates.removeAll(plan.accept(pollUpPredicates, null));
+ return predicates;
+ }
+}
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java
new file mode 100644
index 0000000000..4c8b72ef58
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PredicatePropagation.java
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+
+import com.google.common.collect.Sets;
+
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * derive additional predicates.
+ * for example:
+ * a = b and a = 1 => b = 1
+ */
+public class PredicatePropagation {
+
+ /**
+ * infer additional predicates.
+ */
+ public Set<Expression> infer(Set<Expression> predicates) {
+ Set<Expression> inferred = Sets.newHashSet();
+ for (Expression predicate : predicates) {
+ if (canEquivalentInfer(predicate)) {
+ List<Expression> newInferred = predicates.stream()
+ .filter(p -> !p.equals(predicate))
+ .map(p -> doInfer(predicate, p))
+ .collect(Collectors.toList());
+ inferred.addAll(newInferred);
+ }
+ }
+ inferred.removeAll(predicates);
+ return inferred;
+ }
+
+ /**
+ * Use the left or right child of `leftSlotEqualToRightSlot` to replace the left or right child of `expression`
+ * Now only support infer `ComparisonPredicate`.
+ * TODO: We should determine whether `expression` satisfies the condition for replacement
+ * eg: Satisfy `expression` is non-deterministic
+ */
+ private Expression doInfer(Expression leftSlotEqualToRightSlot, Expression expression) {
+ return expression.accept(new DefaultExpressionRewriter<Void>() {
+
+ @Override
+ public Expression visit(Expression expr, Void context) {
+ return expr;
+ }
+
+ @Override
+ public Expression visitComparisonPredicate(ComparisonPredicate cp, Void context) {
+ if (cp.left().isSlot() && cp.right().isConstant()) {
+ return replaceSlot(cp);
+ } else if (cp.left().isConstant() && cp.right().isSlot()) {
+ return replaceSlot(cp);
+ }
+ return super.visit(cp, context);
+ }
+
+ private Expression replaceSlot(Expression expr) {
+ return expr.rewriteUp(e -> {
+ if (e.equals(leftSlotEqualToRightSlot.child(0))) {
+ return leftSlotEqualToRightSlot.child(1);
+ } else if (e.equals(leftSlotEqualToRightSlot.child(1))) {
+ return leftSlotEqualToRightSlot.child(0);
+ } else {
+ return e;
+ }
+ });
+ }
+ }, null);
+ }
+
+ /**
+ * Currently only equivalence derivation is supported
+ * and requires that the left and right sides of an expression must be slot
+ */
+ private boolean canEquivalentInfer(Expression predicate) {
+ return predicate instanceof EqualTo
+ && predicate.children().stream().allMatch(e -> e instanceof SlotReference)
+ && predicate.child(0).getDataType().equals(predicate.child(1).getDataType());
+ }
+
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PullUpPredicates.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PullUpPredicates.java
new file mode 100644
index 0000000000..9d86415344
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PullUpPredicates.java
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.Collection;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+/**
+ * poll up effective predicates from operator's children.
+ */
+public class PullUpPredicates extends PlanVisitor<ImmutableSet<Expression>, Void> {
+
+ PredicatePropagation propagation = new PredicatePropagation();
+ Map<Plan, ImmutableSet<Expression>> cache = new IdentityHashMap<>();
+
+ @Override
+ public ImmutableSet<Expression> visit(Plan plan, Void context) {
+ if (plan.arity() == 1) {
+ return plan.child(0).accept(this, context);
+ }
+ return ImmutableSet.of();
+ }
+
+ @Override
+ public ImmutableSet<Expression> visitLogicalFilter(LogicalFilter<? extends Plan> filter, Void context) {
+ return cacheOrElse(filter, () -> {
+ List<Expression> predicates = Lists.newArrayList(filter.getConjuncts());
+ predicates.addAll(filter.child().accept(this, context));
+ return getAvailableExpressions(predicates, filter);
+ });
+ }
+
+ @Override
+ public ImmutableSet<Expression> visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, Void context) {
+ return cacheOrElse(join, () -> {
+ Set<Expression> predicates = Sets.newHashSet();
+ ImmutableSet<Expression> leftPredicates = join.left().accept(this, context);
+ ImmutableSet<Expression> rightPredicates = join.right().accept(this, context);
+ switch (join.getJoinType()) {
+ case INNER_JOIN:
+ case CROSS_JOIN:
+ predicates.addAll(leftPredicates);
+ predicates.addAll(rightPredicates);
+ join.getOnClauseCondition().map(on -> predicates.addAll(ExpressionUtils.extractConjunction(on)));
+ break;
+ case LEFT_SEMI_JOIN:
+ predicates.addAll(leftPredicates);
+ join.getOnClauseCondition().map(on -> predicates.addAll(ExpressionUtils.extractConjunction(on)));
+ break;
+ case RIGHT_SEMI_JOIN:
+ predicates.addAll(rightPredicates);
+ join.getOnClauseCondition().map(on -> predicates.addAll(ExpressionUtils.extractConjunction(on)));
+ break;
+ case LEFT_OUTER_JOIN:
+ case LEFT_ANTI_JOIN:
+ predicates.addAll(leftPredicates);
+ break;
+ case RIGHT_OUTER_JOIN:
+ case RIGHT_ANTI_JOIN:
+ predicates.addAll(rightPredicates);
+ break;
+ default:
+ }
+ return getAvailableExpressions(predicates, join);
+ });
+ }
+
+ @Override
+ public ImmutableSet<Expression> visitLogicalProject(LogicalProject<? extends Plan> project, Void context) {
+ return cacheOrElse(project, () -> {
+ ImmutableSet<Expression> childPredicates = project.child().accept(this, context);
+ Map<Expression, Slot> expressionSlotMap = project.getAliasToProducer()
+ .entrySet()
+ .stream()
+ .collect(Collectors.toMap(Entry::getValue, Entry::getKey));
+ Expression expression = ExpressionUtils.replace(ExpressionUtils.and(Lists.newArrayList(childPredicates)),
+ expressionSlotMap);
+ List<Expression> predicates = ExpressionUtils.extractConjunction(expression);
+ return getAvailableExpressions(predicates, project);
+ });
+ }
+
+ @Override
+ public ImmutableSet<Expression> visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, Void context) {
+ return cacheOrElse(aggregate, () -> {
+ ImmutableSet<Expression> childPredicates = aggregate.child().accept(this, context);
+ Map<Expression, Slot> expressionSlotMap = aggregate.getOutputExpressions()
+ .stream()
+ .filter(this::hasAgg)
+ .collect(Collectors.toMap(
+ namedExpr -> {
+ if (namedExpr instanceof Alias) {
+ return ((Alias) namedExpr).child();
+ } else {
+ return namedExpr;
+ }
+ }, NamedExpression::toSlot)
+ );
+ Expression expression = ExpressionUtils.replace(ExpressionUtils.and(Lists.newArrayList(childPredicates)),
+ expressionSlotMap);
+ List<Expression> predicates = ExpressionUtils.extractConjunction(expression);
+ return getAvailableExpressions(predicates, aggregate);
+ });
+ }
+
+ private ImmutableSet<Expression> cacheOrElse(Plan plan, Supplier<ImmutableSet<Expression>> predicatesSupplier) {
+ ImmutableSet<Expression> predicates = cache.get(plan);
+ if (predicates != null) {
+ return predicates;
+ }
+ predicates = predicatesSupplier.get();
+ cache.put(plan, predicates);
+ return predicates;
+ }
+
+ private ImmutableSet<Expression> getAvailableExpressions(Collection<Expression> predicates, Plan plan) {
+ Set<Expression> expressions = Sets.newHashSet(predicates);
+ expressions.addAll(propagation.infer(expressions));
+ return expressions.stream()
+ .filter(p -> plan.getOutputSet().containsAll(p.getInputSlots()))
+ .collect(ImmutableSet.toImmutableSet());
+ }
+
+ private boolean hasAgg(Expression expression) {
+ return expression.anyMatch(AggregateFunction.class::isInstance);
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
index b1d037c282..9ffa29b45e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Expression.java
@@ -139,6 +139,10 @@ public abstract class Expression extends AbstractTreeNode<Expression> implements
return this instanceof NullLiteral;
}
+ public boolean isSlot() {
+ return this instanceof Slot;
+ }
+
@Override
public boolean equals(Object o) {
if (this == o) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java
index d0279b921e..638973d2aa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalJoin.java
@@ -258,4 +258,10 @@ public class LogicalJoin<LEFT_CHILD_TYPE extends Plan, RIGHT_CHILD_TYPE extends
public LogicalJoin withJoinType(JoinType joinType) {
return new LogicalJoin<>(joinType, hashJoinConjuncts, otherJoinConjuncts, left(), right(), joinReorderContext);
}
+
+ public LogicalJoin withOtherJoinConjuncts(List<Expression> otherJoinConjuncts) {
+ return new LogicalJoin<>(joinType, hashJoinConjuncts, otherJoinConjuncts, left(), right(),
+ joinReorderContext);
+ }
}
+
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicatesTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicatesTest.java
new file mode 100644
index 0000000000..3c31b0466c
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/InferPredicatesTest.java
@@ -0,0 +1,531 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.util.PatternMatchSupported;
+import org.apache.doris.nereids.util.PlanChecker;
+import org.apache.doris.utframe.TestWithFeService;
+
+import org.junit.jupiter.api.Test;
+
+public class InferPredicatesTest extends TestWithFeService implements PatternMatchSupported {
+
+ @Override
+ protected void runBeforeAll() throws Exception {
+ createDatabase("test");
+
+ createTable("create table test.student (\n"
+ + "id int not null,\n"
+ + "name varchar(128),\n"
+ + "age int,sex int)\n"
+ + "distributed by hash(id) buckets 10\n"
+ + "properties('replication_num' = '1');");
+
+ createTable("create table test.score (\n"
+ + "sid int not null, \n"
+ + "cid int not null, \n"
+ + "grade double)\n"
+ + "distributed by hash(sid,cid) buckets 10\n"
+ + "properties('replication_num' = '1');");
+
+ createTable("create table test.course (\n"
+ + "id int not null, \n"
+ + "name varchar(128), \n"
+ + "teacher varchar(128))\n"
+ + "distributed by hash(id) buckets 10\n"
+ + "properties('replication_num' = '1');");
+
+ createTables("create table test.subquery1\n"
+ + "(k1 bigint, k2 bigint)\n"
+ + "duplicate key(k1)\n"
+ + "distributed by hash(k2) buckets 1\n"
+ + "properties('replication_num' = '1');\n",
+ "create table test.subquery2\n"
+ + "(k1 varchar(10), k2 bigint)\n"
+ + "partition by range(k2)\n"
+ + "(partition p1 values less than(\"10\"))\n"
+ + "distributed by hash(k2) buckets 1\n"
+ + "properties('replication_num' = '1');",
+ "create table test.subquery3\n"
+ + "(k1 int not null, k2 varchar(128), k3 bigint, v1 bigint, v2 bigint)\n"
+ + "distributed by hash(k2) buckets 1\n"
+ + "properties('replication_num' = '1');",
+ "create table test.subquery4\n"
+ + "(k1 bigint, k2 bigint)\n"
+ + "duplicate key(k1)\n"
+ + "distributed by hash(k2) buckets 1\n"
+ + "properties('replication_num' = '1');");
+
+ connectContext.setDatabase("default_cluster:test");
+ }
+
+ @Test
+ public void inferPredicatesTest01() {
+ String sql = "select * from student join score on student.id = score.sid where student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest02() {
+ String sql = "select * from student join score on student.id = score.sid";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalOlapScan(),
+ logicalOlapScan()
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest03() {
+ String sql = "select * from student join score on student.id = score.sid where student.id in (1,2,3)";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id IN (1, 2, 3)")),
+ logicalOlapScan()
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest04() {
+ String sql = "select * from student join score on student.id = score.sid and student.id in (1,2,3)";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id IN (1, 2, 3)")),
+ logicalOlapScan()
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest05() {
+ String sql = "select * from student join score on student.id = score.sid join course on score.sid = course.id where student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ ),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest06() {
+ String sql = "select * from student join score on student.id = score.sid join course on score.sid = course.id and score.sid > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ ),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest07() {
+ String sql = "select * from student left join score on student.id = score.sid where student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest08() {
+ String sql = "select * from student left join score on student.id = score.sid and student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalOlapScan(),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest09() {
+ // convert left join to inner join
+ String sql = "select * from student left join score on student.id = score.sid where score.sid > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("id > 1")),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest10() {
+ String sql = "select * from (select id as nid, name from student) t left join score on t.nid = score.sid where t.nid > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("id > 1"))
+ ),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest11() {
+ String sql = "select * from (select id as nid, name from student) t left join score on t.nid = score.sid and t.nid > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalProject(
+ logicalOlapScan()
+ ),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest12() {
+ String sql = "select * from student left join (select sid as nid, sum(grade) from score group by sid) s on s.nid = student.id where student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("id > 1")),
+ logicalProject(
+ logicalAggregate(
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ ))
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest13() {
+ String sql = "select * from (select id, name from student where id = 1) t left join score on t.id = score.sid";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("id = 1"))
+ ),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid = 1"))
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest14() {
+ String sql = "select * from student left semi join score on student.id = score.sid where student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest15() {
+ String sql = "select * from student left semi join score on student.id = score.sid and student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filer -> filer.getPredicates().toSql().contains("sid > 1"))
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest16() {
+ String sql = "select * from student left anti join score on student.id = score.sid and student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalOlapScan(),
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest17() {
+ String sql = "select * from student left anti join score on student.id = score.sid and score.sid > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalOlapScan(),
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest18() {
+ String sql = "select * from student left anti join score on student.id = score.sid where student.id > 1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest19() {
+ String sql = "select * from subquery1 left semi join (select t1.k3 from (select * from subquery3 left semi join (select k1 from subquery4 where k1 = 3) t on subquery3.k3 = t.k1) t1 inner join (select k2,sum(k2) as sk2 from subquery2 group by k2) t2 on t2.k2 = t1.v1 and t1.v2 > t2.sk2) t3 on t3.k3 = subquery1.k1";
+ Plan plan = PlanChecker.from(connectContext).analyze(sql).rewrite().getPlan();
+ System.out.println(plan.treeString());
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("k1 = 3")),
+ logicalProject(
+ logicalJoin(
+ logicalJoin(
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("k3 = 3"))
+ ),
+ logicalProject(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("k1 = 3"))
+ )
+ ),
+ logicalProject()
+ )
+ )
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest20() {
+ String sql = "select * from student left join score on student.id = score.sid and score.sid > 1 inner join course on course.id = score.sid";
+ PlanChecker.from(connectContext).analyze(sql).rewrite().printlnTree();
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalJoin(
+ logicalOlapScan(),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ ),
+ logicalOlapScan()
+ )
+ );
+ }
+
+ @Test
+ public void inferPredicatesTest21() {
+ String sql = "select * from student,score,course where student.id = score.sid and score.sid = course.id and score.sid > 1";
+ PlanChecker.from(connectContext).analyze(sql).rewrite().printlnTree();
+ PlanChecker.from(connectContext)
+ .analyze(sql)
+ .rewrite()
+ .matchesFromRoot(
+ logicalJoin(
+ logicalJoin(
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1")),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("sid > 1"))
+ ),
+ logicalFilter(
+ logicalOlapScan()
+ ).when(filter -> filter.getPredicates().toSql().contains("id > 1"))
+ )
+ );
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 25/39: [fix](JSON) Fail to parse JSONPath (libc++) (#13941)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 83663081fecdcbf1846a3bb1a086a1510ebf3321
Author: Adonis Ling <ad...@gmail.com>
AuthorDate: Wed Nov 9 08:58:01 2022 +0800
[fix](JSON) Fail to parse JSONPath (libc++) (#13941)
---
be/src/exprs/json_functions.cpp | 12 +++++++++---
be/src/exprs/json_functions.h | 5 ++---
be/src/util/string_util.h | 13 +++++++------
be/src/vec/functions/function_json.cpp | 8 ++++++--
4 files changed, 24 insertions(+), 14 deletions(-)
diff --git a/be/src/exprs/json_functions.cpp b/be/src/exprs/json_functions.cpp
index b449ffceaf..98756a4d8b 100644
--- a/be/src/exprs/json_functions.cpp
+++ b/be/src/exprs/json_functions.cpp
@@ -299,8 +299,8 @@ rapidjson::Value* JsonFunctions::match_value(const std::vector<JsonPath>& parsed
}
rapidjson::Value* JsonFunctions::get_json_object(FunctionContext* context,
- const std::string_view& json_string,
- const std::string_view& path_string,
+ std::string_view json_string,
+ std::string_view path_string,
const JsonFunctionType& fntype,
rapidjson::Document* document) {
// split path by ".", and escape quota by "\"
@@ -319,13 +319,19 @@ rapidjson::Value* JsonFunctions::get_json_object(FunctionContext* context,
}
if (json_state->json_paths.size() == 0) {
+#ifdef USE_LIBCPP
+ std::string s(path_string);
+ auto tok = get_json_token(s);
+#else
auto tok = get_json_token(path_string);
+#endif
std::vector<std::string> paths(tok.begin(), tok.end());
get_parsed_paths(paths, &json_state->json_paths);
}
#else
json_state = &tmp_json_state;
- auto tok = get_json_token(path_string);
+ std::string s(path_string);
+ auto tok = get_json_token(s);
std::vector<std::string> paths(tok.begin(), tok.end());
get_parsed_paths(paths, &json_state->json_paths);
#endif
diff --git a/be/src/exprs/json_functions.h b/be/src/exprs/json_functions.h
index f976c5ec00..89899271a8 100644
--- a/be/src/exprs/json_functions.h
+++ b/be/src/exprs/json_functions.h
@@ -108,9 +108,8 @@ public:
const doris_udf::StringVal& json_str,
const doris_udf::StringVal& path);
- static rapidjson::Value* get_json_object(FunctionContext* context,
- const std::string_view& json_string,
- const std::string_view& path_string,
+ static rapidjson::Value* get_json_object(FunctionContext* context, std::string_view json_string,
+ std::string_view path_string,
const JsonFunctionType& fntype,
rapidjson::Document* document);
diff --git a/be/src/util/string_util.h b/be/src/util/string_util.h
index 7fce0a8500..0d2fe7e3ee 100644
--- a/be/src/util/string_util.h
+++ b/be/src/util/string_util.h
@@ -130,14 +130,15 @@ template <class T>
using StringCaseUnorderedMap =
std::unordered_map<std::string, T, StringCaseHasher, StringCaseEqual>;
-inline auto get_json_token(const std::string_view& path_string) {
-#ifdef USE_LIBCPP
- return boost::tokenizer<boost::escaped_list_separator<char>>(
- std::string(path_string), boost::escaped_list_separator<char>("\\", ".", "\""));
-#else
+template <typename T>
+inline auto get_json_token(T& path_string) {
return boost::tokenizer<boost::escaped_list_separator<char>>(
path_string, boost::escaped_list_separator<char>("\\", ".", "\""));
-#endif
}
+#ifdef USE_LIBCPP
+template <>
+inline auto get_json_token(std::string_view& path_string) = delete;
+#endif
+
} // namespace doris
diff --git a/be/src/vec/functions/function_json.cpp b/be/src/vec/functions/function_json.cpp
index 55184997a3..d04390be95 100644
--- a/be/src/vec/functions/function_json.cpp
+++ b/be/src/vec/functions/function_json.cpp
@@ -187,13 +187,17 @@ rapidjson::Value* match_value(const std::vector<JsonPath>& parsed_paths, rapidjs
}
template <JsonFunctionType fntype>
-rapidjson::Value* get_json_object(const std::string_view& json_string,
- const std::string_view& path_string,
+rapidjson::Value* get_json_object(std::string_view json_string, std::string_view path_string,
rapidjson::Document* document) {
std::vector<JsonPath>* parsed_paths;
std::vector<JsonPath> tmp_parsed_paths;
+#ifdef USE_LIBCPP
+ std::string s(path_string);
+ auto tok = get_json_token(s);
+#else
auto tok = get_json_token(path_string);
+#endif
std::vector<std::string> paths(tok.begin(), tok.end());
get_parsed_paths(paths, &tmp_parsed_paths);
parsed_paths = &tmp_parsed_paths;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 32/39: [improve](Nereids): remove redundant code, add annotation in Memo. (#14083)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e91dae019806a615cc617ea7ce7b9da37e642017
Author: jakevin <ja...@gmail.com>
AuthorDate: Wed Nov 9 13:39:20 2022 +0800
[improve](Nereids): remove redundant code, add annotation in Memo. (#14083)
---
.../java/org/apache/doris/nereids/memo/Group.java | 69 +---------------------
.../apache/doris/nereids/memo/GroupExpression.java | 25 +++-----
.../java/org/apache/doris/nereids/memo/Memo.java | 52 ++++++++--------
.../java/org/apache/doris/nereids/util/Utils.java | 8 +++
.../org/apache/doris/nereids/memo/MemoTest.java | 8 +--
5 files changed, 50 insertions(+), 112 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
index 978aa0e0fe..686be94c72 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Group.java
@@ -18,12 +18,9 @@
package org.apache.doris.nereids.memo;
import org.apache.doris.common.Pair;
-import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
-import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
-import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
import org.apache.doris.nereids.util.TreeStringUtils;
import org.apache.doris.statistics.StatsDeriveResult;
@@ -46,6 +43,7 @@ import java.util.stream.Collectors;
*/
public class Group {
private final GroupId groupId;
+ // Save all parent GroupExpression to avoid travsing whole Memo.
private final IdentityHashMap<GroupExpression, Void> parentExpressions = new IdentityHashMap<>();
private final List<GroupExpression> logicalExpressions = Lists.newArrayList();
@@ -57,7 +55,7 @@ public class Group {
private final Map<PhysicalProperties, Pair<Double, GroupExpression>> lowestCostPlans = Maps.newHashMap();
private double costLowerBound = -1;
private boolean isExplored = false;
- private boolean hasCost = false;
+
private StatsDeriveResult statistics;
/**
@@ -87,14 +85,6 @@ public class Group {
return groupId;
}
- public boolean isHasCost() {
- return hasCost;
- }
-
- public void setHasCost(boolean hasCost) {
- this.hasCost = hasCost;
- }
-
/**
* Add new {@link GroupExpression} into this group.
*
@@ -132,27 +122,6 @@ public class Group {
logicalExpressions.add(groupExpression);
}
- public void addPhysicalExpression(GroupExpression groupExpression) {
- groupExpression.setOwnerGroup(this);
- physicalExpressions.add(groupExpression);
- }
-
- /**
- * Rewrite the logical group expression to the new logical group expression.
- *
- * @param newExpression new logical group expression
- * @return old logical group expression
- */
- public GroupExpression rewriteLogicalExpression(GroupExpression newExpression,
- LogicalProperties logicalProperties) {
- newExpression.setOwnerGroup(this);
- this.logicalProperties = logicalProperties;
- GroupExpression oldExpression = getLogicalExpression();
- logicalExpressions.clear();
- logicalExpressions.add(newExpression);
- return oldExpression;
- }
-
public List<GroupExpression> clearLogicalExpressions() {
List<GroupExpression> move = logicalExpressions.stream()
.peek(groupExpr -> groupExpr.setOwnerGroup(null))
@@ -173,10 +142,6 @@ public class Group {
return costLowerBound;
}
- public void setCostLowerBound(double costLowerBound) {
- this.costLowerBound = costLowerBound;
- }
-
/**
* Set or update lowestCostPlans: properties --> Pair.of(cost, expression)
*/
@@ -273,32 +238,6 @@ public class Group {
return Optional.ofNullable(lowestCostPlans.get(physicalProperties));
}
- public Map<PhysicalProperties, Pair<Double, GroupExpression>> getLowestCostPlans() {
- return lowestCostPlans;
- }
-
- /**
- * Get the first Plan from Memo.
- */
- public PhysicalPlan extractPlan() throws AnalysisException {
- GroupExpression groupExpression = this.physicalExpressions.get(0);
-
- List<Plan> planChildren = com.google.common.collect.Lists.newArrayList();
- for (int i = 0; i < groupExpression.arity(); i++) {
- planChildren.add(groupExpression.child(i).extractPlan());
- }
-
- Plan plan = groupExpression.getPlan()
- .withChildren(planChildren)
- .withGroupExpression(Optional.of(groupExpression));
- if (!(plan instanceof PhysicalPlan)) {
- throw new AnalysisException("generate logical plan");
- }
- PhysicalPlan physicalPlan = (PhysicalPlan) plan;
-
- return physicalPlan;
- }
-
public List<GroupExpression> getParentGroupExpressions() {
return ImmutableList.copyOf(parentExpressions.keySet());
}
@@ -318,10 +257,6 @@ public class Group {
return parentExpressions.size();
}
- public int parentExpressionNum() {
- return parentExpressions.size();
- }
-
@Override
public boolean equals(Object o) {
if (this == o) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
index a0ad369664..b8e8edcc19 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java
@@ -24,6 +24,7 @@ import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.util.Utils;
import org.apache.doris.statistics.StatsDeriveResult;
import com.google.common.base.Preconditions;
@@ -43,7 +44,7 @@ public class GroupExpression {
private double cost = 0.0;
private CostEstimate costEstimate = null;
private Group ownerGroup;
- private List<Group> children;
+ private final List<Group> children;
private final Plan plan;
private final BitSet ruleMasks;
private boolean statDerived;
@@ -71,11 +72,11 @@ public class GroupExpression {
this.plan = Objects.requireNonNull(plan, "plan can not be null")
.withGroupExpression(Optional.of(this));
this.children = Lists.newArrayList(Objects.requireNonNull(children, "children can not be null"));
+ this.children.forEach(childGroup -> childGroup.addParentExpression(this));
this.ruleMasks = new BitSet(RuleType.SENTINEL.ordinal());
this.statDerived = false;
this.lowestCostTable = Maps.newHashMap();
this.requestPropertiesMap = Maps.newHashMap();
- this.children.forEach(childGroup -> childGroup.addParentExpression(this));
}
public PhysicalProperties getOutputProperties(PhysicalProperties requestProperties) {
@@ -111,21 +112,13 @@ public class GroupExpression {
/**
* replaceChild.
*
- * @param originChild origin child group
+ * @param oldChild origin child group
* @param newChild new child group
*/
- public void replaceChild(Group originChild, Group newChild) {
- originChild.removeParentExpression(this);
- for (int i = 0; i < children.size(); i++) {
- if (children.get(i) == originChild) {
- children.set(i, newChild);
- newChild.addParentExpression(this);
- }
- }
- }
-
- public void setChild(int index, Group group) {
- this.children.set(index, group);
+ public void replaceChild(Group oldChild, Group newChild) {
+ oldChild.removeParentExpression(this);
+ newChild.addParentExpression(this);
+ Utils.replaceList(children, oldChild, newChild);
}
public boolean hasApplied(Rule rule) {
@@ -255,7 +248,7 @@ public class GroupExpression {
if (costEstimate != null) {
builder.append(" est=").append(costEstimate);
}
- builder.append(" (plan=" + plan.toString() + ") children=[");
+ builder.append(" (plan=").append(plan.toString()).append(") children=[");
for (Group group : children) {
builder.append(group.getGroupId()).append(" ");
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
index 81b1dc7c3f..31a5dee6a4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
@@ -72,7 +72,6 @@ public class Memo {
/**
* Add plan to Memo.
- * TODO: add ut later
*
* @param plan {@link Plan} or {@link Expression} to be added
* @param target target group to add node. null to generate new Group
@@ -386,9 +385,10 @@ public class Memo {
}
}
for (GroupExpression groupExpression : needReplaceChild) {
+ // After change GroupExpression children, the hashcode will change,
+ // so need to reinsert into map.
groupExpressions.remove(groupExpression);
List<Group> children = groupExpression.children();
- // TODO: use a better way to replace child, avoid traversing all groupExpression
for (int i = 0; i < children.size(); i++) {
if (children.get(i).equals(source)) {
children.set(i, destination);
@@ -480,6 +480,7 @@ public class Memo {
// case 5:
// if targetGroup is null or targetGroup equal to the existedExpression's ownerGroup,
// then recycle the temporary new group expression
+ // No ownerGroup, don't need ownerGroup.removeChild()
recycleExpression(newExpression);
return CopyInResult.of(false, existedExpression);
}
@@ -512,8 +513,7 @@ public class Memo {
List<GroupExpression> logicalExpressions = fromGroup.clearLogicalExpressions();
recycleGroup(fromGroup);
- recycleLogicalExpressions(targetGroup);
- recyclePhysicalExpressions(targetGroup);
+ recycleLogicalAndPhysicalExpressions(targetGroup);
for (GroupExpression logicalExpression : logicalExpressions) {
targetGroup.addLogicalExpression(logicalExpression);
@@ -522,8 +522,7 @@ public class Memo {
}
private void reInitGroup(Group group, GroupExpression initLogicalExpression, LogicalProperties logicalProperties) {
- recycleLogicalExpressions(group);
- recyclePhysicalExpressions(group);
+ recycleLogicalAndPhysicalExpressions(group);
group.setLogicalProperties(logicalProperties);
group.addLogicalExpression(initLogicalExpression);
@@ -562,42 +561,45 @@ public class Memo {
}
}
+ /**
+ * Notice: this func don't replace { Parent GroupExpressions -> this Group }.
+ */
private void recycleGroup(Group group) {
+ // recycle in memo.
if (groups.get(group.getGroupId()) == group) {
groups.remove(group.getGroupId());
}
- recycleLogicalExpressions(group);
- recyclePhysicalExpressions(group);
- }
- private void recycleLogicalExpressions(Group group) {
- if (!group.getLogicalExpressions().isEmpty()) {
- for (GroupExpression logicalExpression : group.getLogicalExpressions()) {
- recycleExpression(logicalExpression);
- }
- group.clearLogicalExpressions();
- }
+ // recycle children GroupExpression
+ recycleLogicalAndPhysicalExpressions(group);
}
- private void recyclePhysicalExpressions(Group group) {
- if (!group.getPhysicalExpressions().isEmpty()) {
- for (GroupExpression physicalExpression : group.getPhysicalExpressions()) {
- recycleExpression(physicalExpression);
- }
- group.clearPhysicalExpressions();
- }
+ private void recycleLogicalAndPhysicalExpressions(Group group) {
+ group.getLogicalExpressions().forEach(this::recycleExpression);
+ group.clearLogicalExpressions();
+
+ group.getPhysicalExpressions().forEach(this::recycleExpression);
+ group.clearPhysicalExpressions();
}
+ /**
+ * Notice: this func don't clear { OwnerGroup() -> this GroupExpression }.
+ */
private void recycleExpression(GroupExpression groupExpression) {
+ // recycle in memo.
if (groupExpressions.get(groupExpression) == groupExpression) {
groupExpressions.remove(groupExpression);
}
- for (Group childGroup : groupExpression.children()) {
+
+ // recycle parentGroupExpr in childGroup
+ groupExpression.children().forEach(childGroup -> {
// if not any groupExpression reference child group, then recycle the child group
if (childGroup.removeParentExpression(groupExpression) == 0) {
recycleGroup(childGroup);
}
- }
+ });
+
+ groupExpression.setOwnerGroup(null);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
index b4c41bdf3f..927c553964 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/Utils.java
@@ -201,4 +201,12 @@ public class Utils {
public static LocalDateTime getLocalDatetimeFromLong(long dateTime) {
return LocalDateTime.ofInstant(Instant.ofEpochSecond(dateTime), ZoneId.systemDefault());
}
+
+ public static <T> void replaceList(List<T> list, T oldItem, T newItem) {
+ for (int i = 0; i < list.size(); i++) {
+ if (list.get(i) == oldItem) {
+ list.set(i, newItem);
+ }
+ }
+ }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/MemoTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/MemoTest.java
index f9b04d9758..1735ecec1f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/MemoTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/memo/MemoTest.java
@@ -61,13 +61,13 @@ import java.util.Optional;
class MemoTest implements PatternMatchSupported {
- private ConnectContext connectContext = MemoTestUtils.createConnectContext();
+ private final ConnectContext connectContext = MemoTestUtils.createConnectContext();
- private LogicalJoin<LogicalOlapScan, LogicalOlapScan> logicalJoinAB = new LogicalJoin<>(JoinType.INNER_JOIN,
+ private final LogicalJoin<LogicalOlapScan, LogicalOlapScan> logicalJoinAB = new LogicalJoin<>(JoinType.INNER_JOIN,
PlanConstructor.newLogicalOlapScan(0, "A", 0),
PlanConstructor.newLogicalOlapScan(1, "B", 0));
- private LogicalJoin<LogicalJoin<LogicalOlapScan, LogicalOlapScan>, LogicalOlapScan> logicalJoinABC = new LogicalJoin<>(
+ private final LogicalJoin<LogicalJoin<LogicalOlapScan, LogicalOlapScan>, LogicalOlapScan> logicalJoinABC = new LogicalJoin<>(
JoinType.INNER_JOIN, logicalJoinAB, PlanConstructor.newLogicalOlapScan(2, "C", 0));
@Test
@@ -395,7 +395,7 @@ class MemoTest implements PatternMatchSupported {
// valid case: 5 steps
class A extends UnboundRelation {
// 1: declare the Plan has some states
- State state;
+ final State state;
public A(List<String> nameParts, State state) {
this(nameParts, state, Optional.empty());
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 28/39: [fix](Vectorized)fix json_object and json_array function return wrong result on vectorized engine (#13775)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a02be24c40485d93a01103816121aa09b819450e
Author: ChPi <ch...@gmail.com>
AuthorDate: Wed Nov 9 11:26:55 2022 +0800
[fix](Vectorized)fix json_object and json_array function return wrong result on vectorized engine (#13775)
Issue Number: close #13598
---
be/src/vec/functions/function_json.cpp | 92 ++++++++++++++++++----
gensrc/script/doris_builtins_functions.py | 4 +-
.../json_function/test_query_json_array.out | 15 ++++
.../json_function/test_query_json_object.out | 20 +++--
..._object.groovy => test_query_json_array.groovy} | 31 +++++---
.../json_function/test_query_json_object.groovy | 27 ++++---
6 files changed, 142 insertions(+), 47 deletions(-)
diff --git a/be/src/vec/functions/function_json.cpp b/be/src/vec/functions/function_json.cpp
index d04390be95..4014b65610 100644
--- a/be/src/vec/functions/function_json.cpp
+++ b/be/src/vec/functions/function_json.cpp
@@ -438,22 +438,28 @@ struct FunctionJsonArrayImpl {
static void execute_parse(const std::string& type_flags,
const std::vector<const ColumnString*>& data_columns,
std::vector<rapidjson::Value>& objects,
- rapidjson::Document::AllocatorType& allocator) {
+ rapidjson::Document::AllocatorType& allocator,
+ const std::vector<const ColumnUInt8*>& nullmaps) {
for (int i = 0; i < data_columns.size() - 1; i++) {
constexpr_int_match<'0', '5', Reducer>::run(type_flags[i], objects, allocator,
- data_columns[i]);
+ data_columns[i], nullmaps[i]);
}
}
template <typename TypeImpl>
static void execute_type(std::vector<rapidjson::Value>& objects,
rapidjson::Document::AllocatorType& allocator,
- const ColumnString* data_column) {
+ const ColumnString* data_column, const ColumnUInt8* nullmap) {
StringParser::ParseResult result;
rapidjson::Value value;
for (int i = 0; i < objects.size(); i++) {
- TypeImpl::update_value(result, value, data_column->get_data_at(i), allocator);
+ if (nullmap != nullptr && nullmap->get_data()[i]) {
+ JsonParser<'0'>::update_value(result, value, data_column->get_data_at(i),
+ allocator);
+ } else {
+ TypeImpl::update_value(result, value, data_column->get_data_at(i), allocator);
+ }
objects[i].PushBack(value, allocator);
}
}
@@ -468,40 +474,97 @@ struct FunctionJsonObjectImpl {
static void execute_parse(std::string type_flags,
const std::vector<const ColumnString*>& data_columns,
std::vector<rapidjson::Value>& objects,
- rapidjson::Document::AllocatorType& allocator) {
+ rapidjson::Document::AllocatorType& allocator,
+ const std::vector<const ColumnUInt8*>& nullmaps) {
for (auto& array_object : objects) {
array_object.SetObject();
}
for (int i = 0; i + 1 < data_columns.size() - 1; i += 2) {
constexpr_int_match<'0', '5', Reducer>::run(type_flags[i + 1], objects, allocator,
- data_columns[i], data_columns[i + 1]);
+ data_columns[i], data_columns[i + 1],
+ nullmaps[i + 1]);
}
}
template <typename TypeImpl>
static void execute_type(std::vector<rapidjson::Value>& objects,
rapidjson::Document::AllocatorType& allocator,
- const ColumnString* key_column, const ColumnString* value_column) {
+ const ColumnString* key_column, const ColumnString* value_column,
+ const ColumnUInt8* nullmap) {
StringParser::ParseResult result;
rapidjson::Value key;
rapidjson::Value value;
-
for (int i = 0; i < objects.size(); i++) {
JsonParser<'5'>::update_value(result, key, key_column->get_data_at(i),
allocator); // key always is string
- TypeImpl::update_value(result, value, value_column->get_data_at(i), allocator);
+ if (nullmap != nullptr && nullmap->get_data()[i]) {
+ JsonParser<'0'>::update_value(result, value, value_column->get_data_at(i),
+ allocator);
+ } else {
+ TypeImpl::update_value(result, value, value_column->get_data_at(i), allocator);
+ }
objects[i].AddMember(key, value, allocator);
}
}
};
template <typename SpecificImpl>
-struct FunctionJsonImpl {
+class FunctionJsonAlwaysNotNullable : public IFunction {
+public:
static constexpr auto name = SpecificImpl::name;
+ static FunctionPtr create() {
+ return std::make_shared<FunctionJsonAlwaysNotNullable<SpecificImpl>>();
+ }
+
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 0; }
+
+ bool is_variadic() const override { return true; }
+
+ bool use_default_implementation_for_constants() const override { return true; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return std::make_shared<DataTypeString>();
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ size_t result, size_t input_rows_count) override {
+ auto result_column = ColumnString::create();
+
+ std::vector<ColumnPtr> column_ptrs; // prevent converted column destruct
+ std::vector<const ColumnString*> data_columns;
+ std::vector<const ColumnUInt8*> nullmaps;
+ for (int i = 0; i < arguments.size(); i++) {
+ auto column = block.get_by_position(arguments[i]).column;
+ column_ptrs.push_back(column->convert_to_full_column_if_const());
+ const ColumnNullable* col_nullable =
+ check_and_get_column<ColumnNullable>(column_ptrs.back().get());
+ if (col_nullable) {
+ const ColumnUInt8* col_nullmap = check_and_get_column<ColumnUInt8>(
+ col_nullable->get_null_map_column_ptr().get());
+ nullmaps.push_back(col_nullmap);
+ const ColumnString* col = check_and_get_column<ColumnString>(
+ col_nullable->get_nested_column_ptr().get());
+ data_columns.push_back(col);
+ } else {
+ nullmaps.push_back(nullptr);
+ data_columns.push_back(assert_cast<const ColumnString*>(column_ptrs.back().get()));
+ }
+ }
+ execute(data_columns, *assert_cast<ColumnString*>(result_column.get()), input_rows_count,
+ nullmaps);
+ block.get_by_position(result).column = std::move(result_column);
+ return Status::OK();
+ }
+
static void execute(const std::vector<const ColumnString*>& data_columns,
- ColumnString& result_column, size_t input_rows_count) {
+ ColumnString& result_column, size_t input_rows_count,
+ const std::vector<const ColumnUInt8*> nullmaps) {
std::string type_flags = data_columns.back()->get_data_at(0).to_string();
rapidjson::Document document;
@@ -512,13 +575,14 @@ struct FunctionJsonImpl {
objects.emplace_back(rapidjson::kArrayType);
}
- SpecificImpl::execute_parse(type_flags, data_columns, objects, allocator);
+ SpecificImpl::execute_parse(type_flags, data_columns, objects, allocator, nullmaps);
rapidjson::StringBuffer buf;
rapidjson::Writer<rapidjson::StringBuffer> writer(buf);
for (int i = 0; i < input_rows_count; i++) {
buf.Clear();
+ writer.Reset(buf);
objects[i].Accept(writer);
result_column.insert_data(buf.GetString(), buf.GetSize());
}
@@ -596,8 +660,8 @@ void register_function_json(SimpleFunctionFactory& factory) {
factory.register_function<FunctionGetJsonDouble>();
factory.register_function<FunctionGetJsonString>();
- factory.register_function<FunctionJson<FunctionJsonImpl<FunctionJsonArrayImpl>>>();
- factory.register_function<FunctionJson<FunctionJsonImpl<FunctionJsonObjectImpl>>>();
+ factory.register_function<FunctionJsonAlwaysNotNullable<FunctionJsonArrayImpl>>();
+ factory.register_function<FunctionJsonAlwaysNotNullable<FunctionJsonObjectImpl>>();
factory.register_function<FunctionJson<FunctionJsonQuoteImpl>>();
}
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index cc14485d08..1ed177de77 100755
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -2479,10 +2479,10 @@ visible_functions = [
[['json_array'], 'VARCHAR', ['VARCHAR', '...'],
'_ZN5doris13JsonFunctions10json_arrayEPN9doris_udf15FunctionContextEiPKNS1_9StringValE',
- '', '', 'vec', ''],
+ '', '', 'vec', 'ALWAYS_NOT_NULLABLE'],
[['json_object'], 'VARCHAR', ['VARCHAR', '...'],
'_ZN5doris13JsonFunctions11json_objectEPN9doris_udf15FunctionContextEiPKNS1_9StringValE',
- '', '', 'vec', ''],
+ '', '', 'vec', 'ALWAYS_NOT_NULLABLE'],
[['json_quote'], 'VARCHAR', ['VARCHAR'],
'_ZN5doris13JsonFunctions10json_quoteEPN9doris_udf15FunctionContextERKNS1_9StringValE',
'', '', 'vec', ''],
diff --git a/regression-test/data/query/sql_functions/json_function/test_query_json_array.out b/regression-test/data/query/sql_functions/json_function/test_query_json_array.out
new file mode 100644
index 0000000000..bc6c2d0e7f
--- /dev/null
+++ b/regression-test/data/query/sql_functions/json_function/test_query_json_array.out
@@ -0,0 +1,15 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !sql1 --
+["k0",1,"k1",null,"k2",null,"k3",null,"k4",null,"k5",null,"k6","k6"]
+["k0",2,"k1",1,"k2",null,"k3",null,"k4",null,"k5",null,"k6","k6"]
+["k0",3,"k1",null,"k2",true,"k3",null,"k4",null,"k5",null,"k6","k6"]
+["k0",4,"k1",null,"k2",null,"k3","test","k4","2022-01-01 11:11:11","k5",null,"k6","k6"]
+["k0",5,"k1",1,"k2",true,"k3","test","k4","2022-01-01 11:11:11","k5",null,"k6","k6"]
+
+-- !sql2 --
+["k0",1,"k1",null,"k2",null,"k3",null,"k4",null,"k5",null,"k6","k6"]
+["k0",2,"k1",1,"k2",null,"k3",null,"k4",null,"k5",null,"k6","k6"]
+["k0",3,"k1",null,"k2",true,"k3",null,"k4",null,"k5",null,"k6","k6"]
+["k0",4,"k1",null,"k2",null,"k3","test","k4","2022-01-01 11:11:11","k5",null,"k6","k6"]
+["k0",5,"k1",1,"k2",true,"k3","test","k4","2022-01-01 11:11:11","k5",null,"k6","k6"]
+
diff --git a/regression-test/data/query/sql_functions/json_function/test_query_json_object.out b/regression-test/data/query/sql_functions/json_function/test_query_json_object.out
index ee1dd4abd8..f270e8484e 100644
--- a/regression-test/data/query/sql_functions/json_function/test_query_json_object.out
+++ b/regression-test/data/query/sql_functions/json_function/test_query_json_object.out
@@ -1,9 +1,15 @@
-- This file is automatically generated. You should know what you did if you want to edit this
--- !sql --
-{"k1":null}
-{"k1":null}
-{"k1":null}
-{"k1":null}
-{"k1":null}
-{"k1":1}
+-- !sql1 --
+{"k0":1,"k1":null,"k2":null,"k3":null,"k4":null,"k5":null,"k6":"k6"}
+{"k0":2,"k1":1,"k2":null,"k3":null,"k4":null,"k5":null,"k6":"k6"}
+{"k0":3,"k1":null,"k2":true,"k3":null,"k4":null,"k5":null,"k6":"k6"}
+{"k0":4,"k1":null,"k2":null,"k3":"test","k4":"2022-01-01 11:11:11","k5":null,"k6":"k6"}
+{"k0":5,"k1":1,"k2":true,"k3":"test","k4":"2022-01-01 11:11:11","k5":null,"k6":"k6"}
+
+-- !sql2 --
+{"k0":1,"k1":null,"k2":null,"k3":null,"k4":null,"k5":null,"k6":"k6"}
+{"k0":2,"k1":1,"k2":null,"k3":null,"k4":null,"k5":null,"k6":"k6"}
+{"k0":3,"k1":null,"k2":true,"k3":null,"k4":null,"k5":null,"k6":"k6"}
+{"k0":4,"k1":null,"k2":null,"k3":"test","k4":"2022-01-01 11:11:11","k5":null,"k6":"k6"}
+{"k0":5,"k1":1,"k2":true,"k3":"test","k4":"2022-01-01 11:11:11","k5":null,"k6":"k6"}
diff --git a/regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy b/regression-test/suites/query/sql_functions/json_function/test_query_json_array.groovy
similarity index 53%
copy from regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy
copy to regression-test/suites/query/sql_functions/json_function/test_query_json_array.groovy
index 5fae2ac518..6a4ef939a3 100644
--- a/regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy
+++ b/regression-test/suites/query/sql_functions/json_function/test_query_json_array.groovy
@@ -15,29 +15,34 @@
// specific language governing permissions and limitations
// under the License.
-suite("test_query_json_object", "query") {
+suite("test_query_json_array", "query") {
sql "set enable_vectorized_engine = false;"
- def tableName = "test_query_json_object"
+ def tableName = "test_query_json_array"
sql "DROP TABLE IF EXISTS ${tableName}"
sql """
- CREATE TABLE IF NOT EXISTS `${tableName}` (
- `k1` int(11) NULL COMMENT "user id"
+ CREATE TABLE ${tableName} (
+ `k0` int(11) not null,
+ `k1` int(11) NULL,
+ `k2` boolean NULL,
+ `k3` varchar(255),
+ `k4` datetime
) ENGINE=OLAP
- DUPLICATE KEY(`k1`)
+ DUPLICATE KEY(`k0`,`k1`,`k2`,`k3`,`k4`)
COMMENT "OLAP"
- DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+ DISTRIBUTED BY HASH(`k0`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
);
"""
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(1);"
- qt_sql "select json_object(\"k1\",k1) from ${tableName};"
+ sql "insert into ${tableName} values(1,null,null,null,null);"
+ sql "insert into ${tableName} values(2,1,null,null,null);"
+ sql "insert into ${tableName} values(3,null,true,null,null);"
+ sql "insert into ${tableName} values(4,null,null,'test','2022-01-01 11:11:11');"
+ sql "insert into ${tableName} values(5,1,true,'test','2022-01-01 11:11:11');"
+ qt_sql1 "select json_array('k0',k0,'k1',k1,'k2',k2,'k3',k3,'k4',k4,'k5', null,'k6','k6') from ${tableName};"
+ sql "set enable_vectorized_engine = true;"
+ qt_sql2 "select json_array('k0',k0,'k1',k1,'k2',k2,'k3',k3,'k4',k4,'k5', null,'k6','k6') from ${tableName};"
sql "DROP TABLE ${tableName};"
}
diff --git a/regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy b/regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy
index 5fae2ac518..1f35bdf151 100644
--- a/regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy
+++ b/regression-test/suites/query/sql_functions/json_function/test_query_json_object.groovy
@@ -20,24 +20,29 @@ suite("test_query_json_object", "query") {
def tableName = "test_query_json_object"
sql "DROP TABLE IF EXISTS ${tableName}"
sql """
- CREATE TABLE IF NOT EXISTS `${tableName}` (
- `k1` int(11) NULL COMMENT "user id"
+ CREATE TABLE ${tableName} (
+ `k0` int(11) not null,
+ `k1` int(11) NULL,
+ `k2` boolean NULL,
+ `k3` varchar(255),
+ `k4` datetime
) ENGINE=OLAP
- DUPLICATE KEY(`k1`)
+ DUPLICATE KEY(`k0`,`k1`,`k2`,`k3`,`k4`)
COMMENT "OLAP"
- DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+ DISTRIBUTED BY HASH(`k0`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
);
"""
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(null);"
- sql "insert into ${tableName} values(1);"
- qt_sql "select json_object(\"k1\",k1) from ${tableName};"
+ sql "insert into ${tableName} values(1,null,null,null,null);"
+ sql "insert into ${tableName} values(2,1,null,null,null);"
+ sql "insert into ${tableName} values(3,null,true,null,null);"
+ sql "insert into ${tableName} values(4,null,null,'test','2022-01-01 11:11:11');"
+ sql "insert into ${tableName} values(5,1,true,'test','2022-01-01 11:11:11');"
+ qt_sql1 "select json_object('k0',k0,'k1',k1,'k2',k2,'k3',k3,'k4',k4,'k5', null,'k6','k6') from ${tableName};"
+ sql "set enable_vectorized_engine = true;"
+ qt_sql2 "select json_object('k0',k0,'k1',k1,'k2',k2,'k3',k3,'k4',k4,'k5', null,'k6','k6') from ${tableName};"
sql "DROP TABLE ${tableName};"
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 24/39: [typo](docs)update array type doc #14057
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 61049913179794c9c70dff20344350b68c6d8068
Author: Liqf <10...@users.noreply.github.com>
AuthorDate: Wed Nov 9 08:40:38 2022 +0800
[typo](docs)update array type doc #14057
---
docs/en/docs/sql-manual/sql-reference/Data-Types/ARRAY.md | 12 +-----------
.../zh-CN/docs/sql-manual/sql-reference/Data-Types/ARRAY.md | 13 +------------
2 files changed, 2 insertions(+), 23 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-reference/Data-Types/ARRAY.md b/docs/en/docs/sql-manual/sql-reference/Data-Types/ARRAY.md
index bc344e48c6..62d0db7fe5 100644
--- a/docs/en/docs/sql-manual/sql-reference/Data-Types/ARRAY.md
+++ b/docs/en/docs/sql-manual/sql-reference/Data-Types/ARRAY.md
@@ -38,15 +38,6 @@ T-type could be any of:
BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, DECIMAL, DATE,
DATETIME, CHAR, VARCHAR, STRING
```
-### notice
-
-We should turn on the switch for the ARRAY types feature with the following command before use:
-
-```
-$ mysql-client > admin set frontend config("enable_array_type"="true");
-```
-
-In this way the config will be reset after the FE process restarts. For permanent setting, you can add config `enable_array_type=true` inside fe.conf.
### example
@@ -71,9 +62,8 @@ Insert data example:
```
mysql> INSERT INTO `array_test` VALUES (1, [1,2,3,4,5]);
-mysql> INSERT INTO `array_test` VALUES (2, array(6,7,8)), (3, array()), (4, null);
+mysql> INSERT INTO `array_test` VALUES (2, [6,7,8]), (3, []), (4, null);
```
-Note: The above sql supports the array() function only in non-vectorized scenarios, but not in vectorized scenarios.
Select data example:
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Types/ARRAY.md b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Types/ARRAY.md
index 636857e0c5..699d0ba9e4 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Data-Types/ARRAY.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Data-Types/ARRAY.md
@@ -39,16 +39,6 @@ BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, LARGEINT, FLOAT, DOUBLE, DECIMAL, DATE,
DATETIME, CHAR, VARCHAR, STRING
```
-### notice
-
-使用前可以通过如下命令打开Array开关:
-
-```
-$ mysql-client > admin set frontend config("enable_array_type"="true");
-```
-
-这种方式下Array开关会在Fe进程重启后重置,或者在fe.conf中添加`enable_array_type=true`配置项可永久生效。
-
### example
建表示例如下:
@@ -72,9 +62,8 @@ PROPERTIES (
```
mysql> INSERT INTO `array_test` VALUES (1, [1,2,3,4,5]);
-mysql> INSERT INTO `array_test` VALUES (2, array(6,7,8)), (3, array()), (4, null);
+mysql> INSERT INTO `array_test` VALUES (2, [6,7,8]), (3, []), (4, null);
```
-注意:以上sql仅在非向量化场景下,支持 array() 函数,向量化场景不支持。
查询数据示例:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 09/39: [fix](build) fix compile fail on Segment::open (#14058)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8aa048c056bfdc1cad51740cd1f0a1b2c32f2d6c
Author: Pxl <px...@qq.com>
AuthorDate: Tue Nov 8 14:38:40 2022 +0800
[fix](build) fix compile fail on Segment::open (#14058)
---
be/src/olap/in_list_predicate.h | 20 ++++++++++----------
be/src/olap/rowset/beta_rowset_writer.cpp | 4 ++--
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/be/src/olap/in_list_predicate.h b/be/src/olap/in_list_predicate.h
index b85deeb39c..bdadaf4e57 100644
--- a/be/src/olap/in_list_predicate.h
+++ b/be/src/olap/in_list_predicate.h
@@ -402,10 +402,10 @@ private:
auto* nested_col_ptr = vectorized::check_and_get_column<
vectorized::ColumnDictionary<vectorized::Int32>>(column);
auto& data_array = nested_col_ptr->get_data();
- auto& _value_in_dict_flags =
+ auto& value_in_dict_flags =
_segment_id_to_value_in_dict_flags[column->get_rowset_segment_id()];
- if (_value_in_dict_flags.empty()) {
- nested_col_ptr->find_codes(_values, _value_in_dict_flags);
+ if (value_in_dict_flags.empty()) {
+ nested_col_ptr->find_codes(_values, value_in_dict_flags);
}
for (uint16_t i = 0; i < size; i++) {
@@ -420,11 +420,11 @@ private:
}
if constexpr (is_opposite != (PT == PredicateType::IN_LIST)) {
- if (_value_in_dict_flags[data_array[idx]]) {
+ if (value_in_dict_flags[data_array[idx]]) {
sel[new_size++] = idx;
}
} else {
- if (!_value_in_dict_flags[data_array[idx]]) {
+ if (!value_in_dict_flags[data_array[idx]]) {
sel[new_size++] = idx;
}
}
@@ -475,10 +475,10 @@ private:
auto* nested_col_ptr = vectorized::check_and_get_column<
vectorized::ColumnDictionary<vectorized::Int32>>(column);
auto& data_array = nested_col_ptr->get_data();
- auto& _value_in_dict_flags =
+ auto& value_in_dict_flags =
_segment_id_to_value_in_dict_flags[column->get_rowset_segment_id()];
- if (_value_in_dict_flags.empty()) {
- nested_col_ptr->find_codes(_values, _value_in_dict_flags);
+ if (value_in_dict_flags.empty()) {
+ nested_col_ptr->find_codes(_values, value_in_dict_flags);
}
for (uint16_t i = 0; i < size; i++) {
@@ -497,11 +497,11 @@ private:
}
if constexpr (is_opposite != (PT == PredicateType::IN_LIST)) {
- if (is_and ^ _value_in_dict_flags[data_array[idx]]) {
+ if (is_and ^ value_in_dict_flags[data_array[idx]]) {
flags[i] = !is_and;
}
} else {
- if (is_and ^ !_value_in_dict_flags[data_array[idx]]) {
+ if (is_and ^ !value_in_dict_flags[data_array[idx]]) {
flags[i] = !is_and;
}
}
diff --git a/be/src/olap/rowset/beta_rowset_writer.cpp b/be/src/olap/rowset/beta_rowset_writer.cpp
index 795ca8e145..459a7e276d 100644
--- a/be/src/olap/rowset/beta_rowset_writer.cpp
+++ b/be/src/olap/rowset/beta_rowset_writer.cpp
@@ -390,8 +390,8 @@ Status BetaRowsetWriter::_load_noncompacted_segments(
auto cache_path =
BetaRowset::segment_cache_path(_context.rowset_dir, _context.rowset_id, seg_id);
std::shared_ptr<segment_v2::Segment> segment;
- auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, _context.tablet_schema,
- &segment);
+ auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, rowset_id(),
+ _context.tablet_schema, &segment);
if (!s.ok()) {
LOG(WARNING) << "failed to open segment. " << seg_path << ":" << s.to_string();
return Status::OLAPInternalError(OLAP_ERR_ROWSET_LOAD_FAILED);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 18/39: [feature](information_schema) add `backends` information_schema table (#13086)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 81d62d6c935c3c88372a94bc42366403dd262918
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Tue Nov 8 22:15:10 2022 +0800
[feature](information_schema) add `backends` information_schema table (#13086)
---
be/src/exec/CMakeLists.txt | 1 +
be/src/exec/schema_scan_node.cpp | 5 +
be/src/exec/schema_scanner.cpp | 47 +++++-
be/src/exec/schema_scanner.h | 10 ++
.../schema_scanner/schema_backends_scanner.cpp | 165 +++++++++++++++++++++
.../exec/schema_scanner/schema_backends_scanner.h | 44 ++++++
be/src/vec/exec/vschema_scan_node.cpp | 24 ++-
.../org/apache/doris/analysis/SchemaTableType.java | 3 +-
.../java/org/apache/doris/catalog/SchemaTable.java | 51 ++++++-
.../org/apache/doris/planner/SchemaScanNode.java | 2 +
.../apache/doris/service/FrontendServiceImpl.java | 126 ++++++++++++++++
gensrc/thrift/Data.thrift | 9 +-
gensrc/thrift/Descriptors.thrift | 3 +-
gensrc/thrift/FrontendService.thrift | 16 ++
gensrc/thrift/PlanNodes.thrift | 8 +
.../suites/correctness/test_backends_table.groovy | 23 +++
16 files changed, 519 insertions(+), 18 deletions(-)
diff --git a/be/src/exec/CMakeLists.txt b/be/src/exec/CMakeLists.txt
index 1c7f2be313..2e020e2f92 100644
--- a/be/src/exec/CMakeLists.txt
+++ b/be/src/exec/CMakeLists.txt
@@ -88,6 +88,7 @@ set(EXEC_FILES
schema_scanner/schema_files_scanner.cpp
schema_scanner/schema_partitions_scanner.cpp
schema_scanner/schema_rowsets_scanner.cpp
+ schema_scanner/schema_backends_scanner.cpp
partitioned_hash_table.cc
partitioned_aggregation_node.cc
diff --git a/be/src/exec/schema_scan_node.cpp b/be/src/exec/schema_scan_node.cpp
index 6a1b546ec1..e52cddb98f 100644
--- a/be/src/exec/schema_scan_node.cpp
+++ b/be/src/exec/schema_scan_node.cpp
@@ -82,6 +82,11 @@ Status SchemaScanNode::init(const TPlanNode& tnode, RuntimeState* state) {
if (tnode.schema_scan_node.__isset.thread_id) {
_scanner_param.thread_id = tnode.schema_scan_node.thread_id;
}
+
+ if (tnode.schema_scan_node.__isset.table_structure) {
+ _scanner_param.table_structure = _pool->add(
+ new std::vector<TSchemaTableStructure>(tnode.schema_scan_node.table_structure));
+ }
return Status::OK();
}
diff --git a/be/src/exec/schema_scanner.cpp b/be/src/exec/schema_scanner.cpp
index fb4623114c..a452077f6c 100644
--- a/be/src/exec/schema_scanner.cpp
+++ b/be/src/exec/schema_scanner.cpp
@@ -17,6 +17,7 @@
#include "exec/schema_scanner.h"
+#include "exec/schema_scanner/schema_backends_scanner.h"
#include "exec/schema_scanner/schema_charsets_scanner.h"
#include "exec/schema_scanner/schema_collations_scanner.h"
#include "exec/schema_scanner/schema_columns_scanner.h"
@@ -31,6 +32,8 @@
#include "exec/schema_scanner/schema_user_privileges_scanner.h"
#include "exec/schema_scanner/schema_variables_scanner.h"
#include "exec/schema_scanner/schema_views_scanner.h"
+#include "runtime/define_primitive_type.h"
+#include "runtime/string_value.h"
namespace doris {
@@ -41,9 +44,23 @@ SchemaScanner::SchemaScanner(ColumnDesc* columns, int column_num)
_param(nullptr),
_columns(columns),
_column_num(column_num),
- _tuple_desc(nullptr) {}
+ _tuple_desc(nullptr),
+ _schema_table_type(TSchemaTableType::SCH_INVALID) {}
-SchemaScanner::~SchemaScanner() {}
+SchemaScanner::SchemaScanner(ColumnDesc* columns, int column_num, TSchemaTableType::type type)
+ : _is_init(false),
+ _param(nullptr),
+ _columns(columns),
+ _column_num(column_num),
+ _tuple_desc(nullptr),
+ _schema_table_type(type) {}
+
+SchemaScanner::~SchemaScanner() {
+ if (_is_create_columns == true && _columns != nullptr) {
+ delete[] _columns;
+ _columns = nullptr;
+ }
+}
Status SchemaScanner::start(RuntimeState* state) {
if (!_is_init) {
@@ -70,8 +87,15 @@ Status SchemaScanner::init(SchemaScannerParam* param, ObjectPool* pool) {
if (_is_init) {
return Status::OK();
}
+ if (nullptr == param || nullptr == pool) {
+ return Status::InternalError("invalid parameter");
+ }
+
+ if (_schema_table_type == TSchemaTableType::SCH_BACKENDS) {
+ RETURN_IF_ERROR(create_columns(param->table_structure, pool));
+ }
- if (nullptr == param || nullptr == pool || nullptr == _columns) {
+ if (nullptr == _columns) {
return Status::InternalError("invalid parameter");
}
@@ -113,15 +137,30 @@ SchemaScanner* SchemaScanner::create(TSchemaTableType::type type) {
return new (std::nothrow) SchemaPartitionsScanner();
case TSchemaTableType::SCH_ROWSETS:
return new (std::nothrow) SchemaRowsetsScanner();
+ case TSchemaTableType::SCH_BACKENDS:
+ return new (std::nothrow) SchemaBackendsScanner();
default:
return new (std::nothrow) SchemaDummyScanner();
break;
}
}
+Status SchemaScanner::create_columns(const std::vector<TSchemaTableStructure>* table_structure,
+ ObjectPool* pool) {
+ _column_num = table_structure->size();
+ _columns = new ColumnDesc[_column_num];
+ _is_create_columns = true;
+ for (size_t idx = 0; idx < table_structure->size(); ++idx) {
+ _columns[idx].name = table_structure->at(idx).column_name.c_str();
+ _columns[idx].type = thrift_to_type(table_structure->at(idx).type);
+ _columns[idx].size = table_structure->at(idx).len;
+ _columns[idx].is_null = table_structure->at(idx).is_null;
+ }
+ return Status::OK();
+}
+
Status SchemaScanner::create_tuple_desc(ObjectPool* pool) {
int null_column = 0;
-
for (int i = 0; i < _column_num; ++i) {
if (_columns[i].is_null) {
null_column++;
diff --git a/be/src/exec/schema_scanner.h b/be/src/exec/schema_scanner.h
index 2450da24d4..33b73b894c 100644
--- a/be/src/exec/schema_scanner.h
+++ b/be/src/exec/schema_scanner.h
@@ -43,6 +43,7 @@ struct SchemaScannerParam {
const std::string* ip; // frontend ip
int32_t port; // frontend thrift port
int64_t thread_id;
+ const std::vector<TSchemaTableStructure>* table_structure;
SchemaScannerParam()
: db(nullptr),
@@ -68,6 +69,7 @@ public:
int scale = -1;
};
SchemaScanner(ColumnDesc* columns, int column_num);
+ SchemaScanner(ColumnDesc* columns, int column_num, TSchemaTableType::type type);
virtual ~SchemaScanner();
// init object need information, schema etc.
@@ -84,6 +86,8 @@ public:
protected:
Status create_tuple_desc(ObjectPool* pool);
+ Status create_columns(const std::vector<TSchemaTableStructure>* table_structure,
+ ObjectPool* pool);
bool _is_init;
// this is used for sub class
@@ -94,7 +98,13 @@ protected:
int _column_num;
TupleDescriptor* _tuple_desc;
+ // _is_create_columns means if ColumnDesc is created from FE.
+ // `_columns` should be deleted if _is_create_columns = true.
+ bool _is_create_columns = false;
+
static DorisServer* _s_doris_server;
+
+ TSchemaTableType::type _schema_table_type;
};
} // namespace doris
diff --git a/be/src/exec/schema_scanner/schema_backends_scanner.cpp b/be/src/exec/schema_scanner/schema_backends_scanner.cpp
new file mode 100644
index 0000000000..ec29eb60a6
--- /dev/null
+++ b/be/src/exec/schema_scanner/schema_backends_scanner.cpp
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/schema_scanner/schema_backends_scanner.h"
+
+#include <gen_cpp/Descriptors_types.h>
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/HeartbeatService_types.h>
+
+#include "exec/schema_scanner.h"
+#include "gen_cpp/FrontendService.h"
+#include "runtime/client_cache.h"
+#include "runtime/define_primitive_type.h"
+#include "runtime/exec_env.h"
+#include "runtime/primitive_type.h"
+#include "runtime/string_value.h"
+#include "util/thrift_rpc_helper.h"
+
+namespace doris {
+
+SchemaBackendsScanner::SchemaBackendsScanner()
+ : SchemaScanner(nullptr, 0, TSchemaTableType::SCH_BACKENDS), _row_idx(0) {}
+
+Status SchemaBackendsScanner::start(RuntimeState* state) {
+ if (!_is_init) {
+ return Status::InternalError("used before initialized.");
+ }
+ RETURN_IF_ERROR(_fetch_backends_info());
+ RETURN_IF_ERROR(_set_col_name_to_type());
+ return Status::OK();
+}
+
+Status SchemaBackendsScanner::get_next_row(Tuple* tuple, MemPool* pool, bool* eos) {
+ if (!_is_init) {
+ return Status::InternalError("Used before initialized.");
+ }
+ if (nullptr == tuple || nullptr == pool || nullptr == eos) {
+ return Status::InternalError("input pointer is nullptr.");
+ }
+ if (_row_idx >= _batch_data.size()) {
+ *eos = true;
+ return Status::OK();
+ }
+ *eos = false;
+ return _fill_one_row(tuple, pool);
+}
+
+Status SchemaBackendsScanner::_fill_one_row(Tuple* tuple, MemPool* pool) {
+ memset((void*)tuple, 0, _tuple_desc->num_null_bytes());
+ for (size_t col_idx = 0; col_idx < _column_num; ++col_idx) {
+ RETURN_IF_ERROR(_fill_one_col(tuple, pool, col_idx));
+ }
+ ++_row_idx;
+ return Status::OK();
+}
+
+Status SchemaBackendsScanner::_fill_one_col(Tuple* tuple, MemPool* pool, size_t col_idx) {
+ auto it = _col_name_to_type.find(_columns[col_idx].name);
+
+ // if this column is not exist in BE, we fill it with `NULL`.
+ if (it == _col_name_to_type.end()) {
+ if (_columns[col_idx].is_null) {
+ tuple->set_null(_tuple_desc->slots()[col_idx]->null_indicator_offset());
+ } else {
+ return Status::InternalError("column {} is not found in BE, and {} is not nullable.",
+ _columns[col_idx].name, _columns[col_idx].name);
+ }
+ } else if (it->second == TYPE_BIGINT) {
+ void* slot = tuple->get_slot(_tuple_desc->slots()[col_idx]->tuple_offset());
+ *(reinterpret_cast<int64_t*>(slot)) = _batch_data[_row_idx].column_value[col_idx].longVal;
+ } else if (it->second == TYPE_INT) {
+ void* slot = tuple->get_slot(_tuple_desc->slots()[col_idx]->tuple_offset());
+ *(reinterpret_cast<int32_t*>(slot)) = _batch_data[_row_idx].column_value[col_idx].intVal;
+ } else if (it->second == TYPE_VARCHAR) {
+ void* slot = tuple->get_slot(_tuple_desc->slots()[col_idx]->tuple_offset());
+ StringValue* str_slot = reinterpret_cast<StringValue*>(slot);
+ str_slot->ptr =
+ (char*)pool->allocate(_batch_data[_row_idx].column_value[col_idx].stringVal.size());
+ str_slot->len = _batch_data[_row_idx].column_value[col_idx].stringVal.size();
+ memcpy(str_slot->ptr, _batch_data[_row_idx].column_value[col_idx].stringVal.c_str(),
+ str_slot->len);
+ } else if (it->second == TYPE_DOUBLE) {
+ void* slot = tuple->get_slot(_tuple_desc->slots()[col_idx]->tuple_offset());
+ *(reinterpret_cast<double_t*>(slot)) =
+ _batch_data[_row_idx].column_value[col_idx].doubleVal;
+ } else {
+ // other type
+ }
+ return Status::OK();
+}
+
+Status SchemaBackendsScanner::_fetch_backends_info() {
+ TFetchSchemaTableDataRequest request;
+ request.cluster_name = "";
+ request.__isset.cluster_name = true;
+ request.schema_table_name = TSchemaTableName::BACKENDS;
+ request.__isset.schema_table_name = true;
+ TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
+ // TODO(ftw): if result will too large?
+ TFetchSchemaTableDataResult result;
+
+ RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
+ master_addr.hostname, master_addr.port,
+ [&request, &result](FrontendServiceConnection& client) {
+ client->fetchSchemaTableData(result, request);
+ },
+ config::txn_commit_rpc_timeout_ms));
+
+ Status status(result.status);
+ if (!status.ok()) {
+ LOG(WARNING) << "fetch schema table data from master failed, errmsg="
+ << status.get_error_msg();
+ return status;
+ }
+ _batch_data = std::move(result.data_batch);
+ return Status::OK();
+}
+
+Status SchemaBackendsScanner::_set_col_name_to_type() {
+ _col_name_to_type.emplace("BackendId", TYPE_BIGINT);
+ _col_name_to_type.emplace("TabletNum", TYPE_BIGINT);
+
+ _col_name_to_type.emplace("HeartbeatPort", TYPE_INT);
+ _col_name_to_type.emplace("BePort", TYPE_INT);
+ _col_name_to_type.emplace("HttpPort", TYPE_INT);
+ _col_name_to_type.emplace("BrpcPort", TYPE_INT);
+
+ _col_name_to_type.emplace("Cluster", TYPE_VARCHAR);
+ _col_name_to_type.emplace("IP", TYPE_VARCHAR);
+ _col_name_to_type.emplace("LastStartTime", TYPE_VARCHAR);
+ _col_name_to_type.emplace("LastHeartbeat", TYPE_VARCHAR);
+ _col_name_to_type.emplace("Alive", TYPE_VARCHAR);
+ _col_name_to_type.emplace("SystemDecommissioned", TYPE_VARCHAR);
+ _col_name_to_type.emplace("ClusterDecommissioned", TYPE_VARCHAR);
+
+ _col_name_to_type.emplace("DataUsedCapacity", TYPE_BIGINT);
+ _col_name_to_type.emplace("AvailCapacity", TYPE_BIGINT);
+ _col_name_to_type.emplace("TotalCapacity", TYPE_BIGINT);
+
+ _col_name_to_type.emplace("UsedPct", TYPE_DOUBLE);
+ _col_name_to_type.emplace("MaxDiskUsedPct", TYPE_DOUBLE);
+
+ _col_name_to_type.emplace("RemoteUsedCapacity", TYPE_BIGINT);
+
+ _col_name_to_type.emplace("Tag", TYPE_VARCHAR);
+ _col_name_to_type.emplace("ErrMsg", TYPE_VARCHAR);
+ _col_name_to_type.emplace("Version", TYPE_VARCHAR);
+ _col_name_to_type.emplace("Status", TYPE_VARCHAR);
+ return Status::OK();
+}
+} // namespace doris
diff --git a/be/src/exec/schema_scanner/schema_backends_scanner.h b/be/src/exec/schema_scanner/schema_backends_scanner.h
new file mode 100644
index 0000000000..32753f568e
--- /dev/null
+++ b/be/src/exec/schema_scanner/schema_backends_scanner.h
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "exec/schema_scanner.h"
+namespace doris {
+
+class SchemaBackendsScanner : public SchemaScanner {
+public:
+ SchemaBackendsScanner();
+ ~SchemaBackendsScanner() override = default;
+
+ Status start(RuntimeState* state) override;
+ Status get_next_row(Tuple* tuple, MemPool* pool, bool* eos) override;
+
+private:
+ Status _fill_one_row(Tuple* tuple, MemPool* pool);
+ Status _fetch_backends_info();
+ Status _fill_one_col(Tuple* tuple, MemPool* pool, size_t idx);
+ Status _set_col_name_to_type();
+
+private:
+ // column_name -> type, set by _set_col_name_to_type()
+ std::unordered_map<std::string, PrimitiveType> _col_name_to_type;
+
+ std::vector<TRow> _batch_data;
+ size_t _row_idx;
+};
+} // namespace doris
diff --git a/be/src/vec/exec/vschema_scan_node.cpp b/be/src/vec/exec/vschema_scan_node.cpp
index dfb6811cbf..0f2710b665 100644
--- a/be/src/vec/exec/vschema_scan_node.cpp
+++ b/be/src/vec/exec/vschema_scan_node.cpp
@@ -50,10 +50,10 @@ VSchemaScanNode::~VSchemaScanNode() {
_src_tuple = nullptr;
delete[] reinterpret_cast<char*>(_src_single_tuple);
- _src_single_tuple = NULL;
+ _src_single_tuple = nullptr;
delete[] reinterpret_cast<char*>(_dest_single_tuple);
- _dest_single_tuple = NULL;
+ _dest_single_tuple = nullptr;
}
Status VSchemaScanNode::init(const TPlanNode& tnode, RuntimeState* state) {
@@ -92,6 +92,11 @@ Status VSchemaScanNode::init(const TPlanNode& tnode, RuntimeState* state) {
if (tnode.schema_scan_node.__isset.thread_id) {
_scanner_param.thread_id = tnode.schema_scan_node.thread_id;
}
+
+ if (tnode.schema_scan_node.__isset.table_structure) {
+ _scanner_param.table_structure = _pool->add(
+ new std::vector<TSchemaTableStructure>(tnode.schema_scan_node.table_structure));
+ }
return Status::OK();
}
@@ -138,7 +143,7 @@ Status VSchemaScanNode::prepare(RuntimeState* state) {
// new one mem pool
_tuple_pool.reset(new (std::nothrow) MemPool());
- if (nullptr == _tuple_pool.get()) {
+ if (nullptr == _tuple_pool) {
return Status::InternalError("Allocate MemPool failed.");
}
@@ -161,7 +166,7 @@ Status VSchemaScanNode::prepare(RuntimeState* state) {
// new one scanner
_schema_scanner.reset(SchemaScanner::create(schema_table->schema_table_type()));
- if (nullptr == _schema_scanner.get()) {
+ if (nullptr == _schema_scanner) {
return Status::InternalError("schema scanner get nullptr pointer.");
}
@@ -221,13 +226,13 @@ Status VSchemaScanNode::prepare(RuntimeState* state) {
_src_single_tuple =
reinterpret_cast<doris::Tuple*>(new (std::nothrow) char[_src_tuple_desc->byte_size()]);
- if (NULL == _src_single_tuple) {
+ if (nullptr == _src_single_tuple) {
return Status::InternalError("new src single tuple failed.");
}
_dest_single_tuple =
reinterpret_cast<doris::Tuple*>(new (std::nothrow) char[_dest_tuple_desc->byte_size()]);
- if (NULL == _dest_single_tuple) {
+ if (nullptr == _dest_single_tuple) {
return Status::InternalError("new desc single tuple failed.");
}
@@ -239,9 +244,12 @@ Status VSchemaScanNode::get_next(RuntimeState* state, vectorized::Block* block,
SCOPED_TIMER(_runtime_profile->total_time_counter());
VLOG_CRITICAL << "VSchemaScanNode::GetNext";
- if (state == NULL || block == NULL || eos == NULL)
+ if (state == nullptr || block == nullptr || eos == nullptr) {
return Status::InternalError("input is NULL pointer");
- if (!_is_init) return Status::InternalError("used before initialize.");
+ }
+ if (!_is_init) {
+ return Status::InternalError("used before initialize.");
+ }
RETURN_IF_CANCELLED(state);
std::vector<vectorized::MutableColumnPtr> columns(_slot_num);
bool schema_eos = false;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
index 43d771953f..c8df72b7de 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
@@ -66,7 +66,8 @@ public enum SchemaTableType {
SCH_VIEWS("VIEWS", "VIEWS", TSchemaTableType.SCH_VIEWS),
SCH_CREATE_TABLE("CREATE_TABLE", "CREATE_TABLE", TSchemaTableType.SCH_CREATE_TABLE),
SCH_INVALID("NULL", "NULL", TSchemaTableType.SCH_INVALID),
- SCH_ROWSETS("ROWSETS", "ROWSETS", TSchemaTableType.SCH_ROWSETS);
+ SCH_ROWSETS("ROWSETS", "ROWSETS", TSchemaTableType.SCH_ROWSETS),
+ SCH_BACKENDS("BACKENDS", "BACKENDS", TSchemaTableType.SCH_BACKENDS);
private static final String dbName = "INFORMATION_SCHEMA";
private static SelectList fullSelectLists;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
index 6ebc85f02c..7e06a00f33 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
@@ -20,6 +20,7 @@ package org.apache.doris.catalog;
import org.apache.doris.analysis.SchemaTableType;
import org.apache.doris.common.SystemIdGenerator;
import org.apache.doris.thrift.TSchemaTable;
+import org.apache.doris.thrift.TSchemaTableStructure;
import org.apache.doris.thrift.TTableDescriptor;
import org.apache.doris.thrift.TTableType;
@@ -399,8 +400,54 @@ public class SchemaTable extends Table {
.column("CREATION_TIME", ScalarType.createType(PrimitiveType.BIGINT))
.column("OLDEST_WRITE_TIMESTAMP", ScalarType.createType(PrimitiveType.BIGINT))
.column("NEWEST_WRITE_TIMESTAMP", ScalarType.createType(PrimitiveType.BIGINT))
- .build())).build();
- private SchemaTableType schemaTableType;
+ .build()))
+ .put("backends", new SchemaTable(SystemIdGenerator.getNextId(), "backends", TableType.SCHEMA,
+ builder().column("BackendId", ScalarType.createType(PrimitiveType.BIGINT))
+ .column("Cluster", ScalarType.createVarchar(64))
+ .column("IP", ScalarType.createVarchar(16))
+ .column("HeartbeatPort", ScalarType.createType(PrimitiveType.INT))
+ .column("BePort", ScalarType.createType(PrimitiveType.INT))
+ .column("HttpPort", ScalarType.createType(PrimitiveType.INT))
+ .column("BrpcPort", ScalarType.createType(PrimitiveType.INT))
+ .column("LastStartTime", ScalarType.createVarchar(32))
+ .column("LastHeartbeat", ScalarType.createVarchar(32))
+ .column("Alive", ScalarType.createVarchar(8))
+ .column("SystemDecommissioned", ScalarType.createVarchar(8))
+ .column("ClusterDecommissioned", ScalarType.createVarchar(8))
+ .column("TabletNum", ScalarType.createType(PrimitiveType.BIGINT))
+ .column("DataUsedCapacity", ScalarType.createType(PrimitiveType.BIGINT))
+ .column("AvailCapacity", ScalarType.createType(PrimitiveType.BIGINT))
+ .column("TotalCapacity", ScalarType.createType(PrimitiveType.BIGINT))
+ .column("UsedPct", ScalarType.createType(PrimitiveType.DOUBLE))
+ .column("MaxDiskUsedPct", ScalarType.createType(PrimitiveType.DOUBLE))
+ .column("RemoteUsedCapacity", ScalarType.createType(PrimitiveType.BIGINT))
+ .column("Tag", ScalarType.createVarchar(128))
+ .column("ErrMsg", ScalarType.createVarchar(2048))
+ .column("Version", ScalarType.createVarchar(64))
+ .column("Status", ScalarType.createVarchar(1024))
+ .build()))
+ .build();
+
+ public static List<TSchemaTableStructure> getTableStructure(String tableName) {
+ List<TSchemaTableStructure> tSchemaTableStructureList = Lists.newArrayList();
+ switch (tableName) {
+ case "backends": {
+ Table table = TABLE_MAP.get(tableName);
+ for (Column column : table.getFullSchema()) {
+ TSchemaTableStructure tSchemaTableStructure = new TSchemaTableStructure();
+ tSchemaTableStructure.setColumnName(column.getName());
+ tSchemaTableStructure.setType(column.getDataType().toThrift());
+ tSchemaTableStructure.setLen(column.getDataType().getSlotSize());
+ tSchemaTableStructure.setIsNull(column.isAllowNull());
+ tSchemaTableStructureList.add(tSchemaTableStructure);
+ }
+ break;
+ }
+ default:
+ break;
+ }
+ return tSchemaTableStructureList;
+ }
protected SchemaTable(long id, String name, TableType type, List<Column> baseSchema) {
super(id, name, type, baseSchema);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/SchemaScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/SchemaScanNode.java
index db70e0c623..4b5f5001b4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SchemaScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SchemaScanNode.java
@@ -110,6 +110,8 @@ public class SchemaScanNode extends ScanNode {
TUserIdentity tCurrentUser = ConnectContext.get().getCurrentUserIdentity().toThrift();
msg.schema_scan_node.setCurrentUserIdent(tCurrentUser);
+
+ msg.schema_scan_node.setTableStructure(SchemaTable.getTableStructure(tableName));
}
/**
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index 53ee7dd637..57a7cb58d8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -17,6 +17,7 @@
package org.apache.doris.service;
+import org.apache.doris.alter.DecommissionType;
import org.apache.doris.analysis.SetType;
import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.catalog.Column;
@@ -30,6 +31,7 @@ import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.catalog.external.ExternalDatabase;
import org.apache.doris.catalog.external.ExternalTable;
+import org.apache.doris.cluster.Cluster;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.AuthenticationException;
@@ -43,6 +45,7 @@ import org.apache.doris.common.ThriftServerContext;
import org.apache.doris.common.ThriftServerEventProcessor;
import org.apache.doris.common.UserException;
import org.apache.doris.common.Version;
+import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.ExternalCatalog;
import org.apache.doris.datasource.InternalCatalog;
@@ -57,11 +60,13 @@ import org.apache.doris.qe.ConnectProcessor;
import org.apache.doris.qe.QeProcessorImpl;
import org.apache.doris.qe.QueryState;
import org.apache.doris.qe.VariableMgr;
+import org.apache.doris.system.Backend;
import org.apache.doris.system.Frontend;
import org.apache.doris.system.SystemInfoService;
import org.apache.doris.task.StreamLoadTask;
import org.apache.doris.thrift.FrontendService;
import org.apache.doris.thrift.FrontendServiceVersion;
+import org.apache.doris.thrift.TCell;
import org.apache.doris.thrift.TColumnDef;
import org.apache.doris.thrift.TColumnDesc;
import org.apache.doris.thrift.TDescribeTableParams;
@@ -69,6 +74,8 @@ import org.apache.doris.thrift.TDescribeTableResult;
import org.apache.doris.thrift.TExecPlanFragmentParams;
import org.apache.doris.thrift.TFeResult;
import org.apache.doris.thrift.TFetchResourceResult;
+import org.apache.doris.thrift.TFetchSchemaTableDataRequest;
+import org.apache.doris.thrift.TFetchSchemaTableDataResult;
import org.apache.doris.thrift.TFinishTaskRequest;
import org.apache.doris.thrift.TFrontendPingFrontendRequest;
import org.apache.doris.thrift.TFrontendPingFrontendResult;
@@ -99,6 +106,7 @@ import org.apache.doris.thrift.TPrivilegeStatus;
import org.apache.doris.thrift.TReportExecStatusParams;
import org.apache.doris.thrift.TReportExecStatusResult;
import org.apache.doris.thrift.TReportRequest;
+import org.apache.doris.thrift.TRow;
import org.apache.doris.thrift.TS3StorageParam;
import org.apache.doris.thrift.TShowVariableRequest;
import org.apache.doris.thrift.TShowVariableResult;
@@ -119,9 +127,11 @@ import org.apache.doris.transaction.TransactionState.TxnSourceType;
import org.apache.doris.transaction.TxnCommitAttachment;
import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
+import com.google.gson.Gson;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.thrift.TException;
@@ -977,6 +987,122 @@ public class FrontendServiceImpl implements FrontendService.Iface {
return result;
}
+ @Override
+ public TFetchSchemaTableDataResult fetchSchemaTableData(TFetchSchemaTableDataRequest request) throws TException {
+ switch (request.getSchemaTableName()) {
+ case BACKENDS:
+ return getBackendsSchemaTable(request);
+ default:
+ break;
+ }
+ TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult();
+ result.setStatus(new TStatus(TStatusCode.INTERNAL_ERROR));
+ return result;
+ }
+
+ private TFetchSchemaTableDataResult getBackendsSchemaTable(TFetchSchemaTableDataRequest request) {
+ final SystemInfoService clusterInfoService = Env.getCurrentSystemInfo();
+ TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult();
+ List<Long> backendIds = null;
+ if (!Strings.isNullOrEmpty(request.cluster_name)) {
+ final Cluster cluster = Env.getCurrentEnv().getCluster(request.cluster_name);
+ // root not in any cluster
+ if (null == cluster) {
+ return result;
+ }
+ backendIds = cluster.getBackendIdList();
+ } else {
+ backendIds = clusterInfoService.getBackendIds(false);
+ if (backendIds == null) {
+ return result;
+ }
+ }
+
+ long start = System.currentTimeMillis();
+ Stopwatch watch = Stopwatch.createUnstarted();
+
+ List<TRow> dataBatch = Lists.newArrayList();
+ for (long backendId : backendIds) {
+ Backend backend = clusterInfoService.getBackend(backendId);
+ if (backend == null) {
+ continue;
+ }
+
+ watch.start();
+ Integer tabletNum = Env.getCurrentInvertedIndex().getTabletNumByBackendId(backendId);
+ watch.stop();
+
+ TRow trow = new TRow();
+ trow.addToColumnValue(new TCell().setLongVal(backendId));
+ trow.addToColumnValue(new TCell().setStringVal(backend.getOwnerClusterName()));
+ trow.addToColumnValue(new TCell().setStringVal(backend.getHost()));
+ if (Strings.isNullOrEmpty(request.cluster_name)) {
+ trow.addToColumnValue(new TCell().setIntVal(backend.getHeartbeatPort()));
+ trow.addToColumnValue(new TCell().setIntVal(backend.getBePort()));
+ trow.addToColumnValue(new TCell().setIntVal(backend.getHttpPort()));
+ trow.addToColumnValue(new TCell().setIntVal(backend.getBrpcPort()));
+ }
+ trow.addToColumnValue(new TCell().setStringVal(TimeUtils.longToTimeString(backend.getLastStartTime())));
+ trow.addToColumnValue(new TCell().setStringVal(TimeUtils.longToTimeString(backend.getLastUpdateMs())));
+ trow.addToColumnValue(new TCell().setStringVal(String.valueOf(backend.isAlive())));
+ if (backend.isDecommissioned() && backend.getDecommissionType() == DecommissionType.ClusterDecommission) {
+ trow.addToColumnValue(new TCell().setStringVal("false"));
+ trow.addToColumnValue(new TCell().setStringVal("true"));
+ } else if (backend.isDecommissioned()
+ && backend.getDecommissionType() == DecommissionType.SystemDecommission) {
+ trow.addToColumnValue(new TCell().setStringVal("true"));
+ trow.addToColumnValue(new TCell().setStringVal("false"));
+ } else {
+ trow.addToColumnValue(new TCell().setStringVal("false"));
+ trow.addToColumnValue(new TCell().setStringVal("false"));
+ }
+ trow.addToColumnValue(new TCell().setLongVal(tabletNum));
+
+ // capacity
+ // data used
+ trow.addToColumnValue(new TCell().setLongVal(backend.getDataUsedCapacityB()));
+
+ // available
+ long availB = backend.getAvailableCapacityB();
+ trow.addToColumnValue(new TCell().setLongVal(availB));
+
+ // total
+ long totalB = backend.getTotalCapacityB();
+ trow.addToColumnValue(new TCell().setLongVal(totalB));
+
+ // used percent
+ double used = 0.0;
+ if (totalB <= 0) {
+ used = 0.0;
+ } else {
+ used = (double) (totalB - availB) * 100 / totalB;
+ }
+ trow.addToColumnValue(new TCell().setDoubleVal(used));
+ trow.addToColumnValue(new TCell().setDoubleVal(backend.getMaxDiskUsedPct() * 100));
+
+ // remote used capacity
+ trow.addToColumnValue(new TCell().setLongVal(backend.getRemoteUsedCapacityB()));
+
+ // tags
+ trow.addToColumnValue(new TCell().setStringVal(backend.getTagMapString()));
+ // err msg
+ trow.addToColumnValue(new TCell().setStringVal(backend.getHeartbeatErrMsg()));
+ // version
+ trow.addToColumnValue(new TCell().setStringVal(backend.getVersion()));
+ // status
+ trow.addToColumnValue(new TCell().setStringVal(new Gson().toJson(backend.getBackendStatus())));
+ dataBatch.add(trow);
+ }
+
+ // backends proc node get result too slow, add log to observer.
+ LOG.debug("backends proc get tablet num cost: {}, total cost: {}",
+ watch.elapsed(TimeUnit.MILLISECONDS), (System.currentTimeMillis() - start));
+
+ result.setDataBatch(dataBatch);
+ result.setStatus(new TStatus(TStatusCode.OK));
+ return result;
+ }
+
private TNetworkAddress getClientAddr() {
ThriftServerContext connectionContext = ThriftServerEventProcessor.getConnectionContext();
// For NonBlockingServer, we can not get client ip.
diff --git a/gensrc/thrift/Data.thrift b/gensrc/thrift/Data.thrift
index 676e22a0f2..fee713694f 100644
--- a/gensrc/thrift/Data.thrift
+++ b/gensrc/thrift/Data.thrift
@@ -47,17 +47,22 @@ struct TRowBatch {
}
// this is a union over all possible return types
-struct TColumnValue {
+struct TCell {
// TODO: use <type>_val instead of camelcase
1: optional bool boolVal
2: optional i32 intVal
3: optional i64 longVal
4: optional double doubleVal
5: optional string stringVal
+ // add type: date datetime
}
struct TResultRow {
- 1: list<TColumnValue> colVals
+ 1: list<TCell> colVals
+}
+
+struct TRow {
+ 1: optional list<TCell> column_value
}
// Serialized, self-contained version of a RowBatch (in be/src/runtime/row-batch.h).
diff --git a/gensrc/thrift/Descriptors.thrift b/gensrc/thrift/Descriptors.thrift
index f9517e2bfa..a3fc84571b 100644
--- a/gensrc/thrift/Descriptors.thrift
+++ b/gensrc/thrift/Descriptors.thrift
@@ -103,7 +103,8 @@ enum TSchemaTableType {
SCH_VARIABLES,
SCH_VIEWS,
SCH_INVALID,
- SCH_ROWSETS
+ SCH_ROWSETS,
+ SCH_BACKENDS
}
enum THdfsCompression {
diff --git a/gensrc/thrift/FrontendService.thrift b/gensrc/thrift/FrontendService.thrift
index 5d48c1ae7e..9cf6ffb6ee 100644
--- a/gensrc/thrift/FrontendService.thrift
+++ b/gensrc/thrift/FrontendService.thrift
@@ -689,6 +689,20 @@ struct TInitExternalCtlMetaResult {
2: optional string status;
}
+enum TSchemaTableName{
+ BACKENDS = 0,
+}
+
+struct TFetchSchemaTableDataRequest {
+ 1: optional string cluster_name
+ 2: optional TSchemaTableName schema_table_name
+}
+
+struct TFetchSchemaTableDataResult {
+ 1: required Status.TStatus status
+ 2: optional list<Data.TRow> data_batch;
+}
+
service FrontendService {
TGetDbsResult getDbNames(1: TGetDbsParams params)
TGetTablesResult getTableNames(1: TGetTablesParams params)
@@ -725,4 +739,6 @@ service FrontendService {
AgentService.TGetStoragePolicyResult refreshStoragePolicy()
TInitExternalCtlMetaResult initExternalCtlMeta(1: TInitExternalCtlMetaRequest request)
+
+ TFetchSchemaTableDataResult fetchSchemaTableData(1: TFetchSchemaTableDataRequest request)
}
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 2df3fea6a2..f8566c2b5d 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -453,6 +453,13 @@ struct TCsvScanNode {
10:optional map<string, TMiniLoadEtlFunction> column_function_mapping
}
+struct TSchemaTableStructure {
+ 1: optional string column_name
+ 2: optional Types.TPrimitiveType type
+ 3: optional i64 len
+ 4: optional bool is_null;
+}
+
struct TSchemaScanNode {
1: required Types.TTupleId tuple_id
@@ -467,6 +474,7 @@ struct TSchemaScanNode {
10: optional string user_ip // deprecated
11: optional Types.TUserIdentity current_user_ident // to replace the user and user_ip
12: optional bool show_hidden_cloumns = false
+ 13: optional list<TSchemaTableStructure> table_structure
}
struct TMetaScanNode {
diff --git a/regression-test/suites/correctness/test_backends_table.groovy b/regression-test/suites/correctness/test_backends_table.groovy
new file mode 100644
index 0000000000..3b4771d825
--- /dev/null
+++ b/regression-test/suites/correctness/test_backends_table.groovy
@@ -0,0 +1,23 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// This suit test the `backends` information_schema table
+suite("test_backends_table") {
+ List<List<Object>> table = sql """ select * from information_schema.backends; """
+ assertTrue(table.size() > 0) // row should > 0
+ assertTrue(table[0].size == 23) // column should be 23
+}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 31/39: [feature](Nereids) binding slot in order by that not show in project (#14042)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0ef2377d220b73e3a17b0f36517123a6dc82e035
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Wed Nov 9 13:25:41 2022 +0800
[feature](Nereids) binding slot in order by that not show in project (#14042)
1. binding slot in order by that not show in project, such as:
SELECT c1 FROM t WHERE c2 > 0 ORDER BY c3
2. not check unbound when bind slot reference. Instead, do it in analysis check.
---
.../doris/nereids/jobs/batch/AnalyzeRulesJob.java | 4 +--
.../nereids/processor/post/PlanPostProcessors.java | 2 +-
.../org/apache/doris/nereids/rules/RuleType.java | 7 +++--
.../nereids/rules/analysis/BindSlotReference.java | 11 ++++++--
.../nereids/rules/analysis/CheckAnalysis.java | 21 +++++++++++++-
...egateFunctions.java => FillUpMissingSlots.java} | 31 ++++++++++++++++++---
.../datasets/clickbench/AnalyzeClickBenchTest.java | 16 +++++------
.../nereids/datasets/tpch/AnalyzeTPCHTest.java | 5 ++--
.../doris/nereids/datasets/tpch/TPCHUtils.java | 32 +++++++++++-----------
...ctionsTest.java => FillUpMissingSlotsTest.java} | 2 +-
.../doris/nereids/util/LogicalPlanBuilder.java | 17 ++++--------
11 files changed, 95 insertions(+), 53 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/AnalyzeRulesJob.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/AnalyzeRulesJob.java
index ce5c589a80..db7528d4bc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/AnalyzeRulesJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/AnalyzeRulesJob.java
@@ -21,9 +21,9 @@ import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.rules.analysis.BindFunction;
import org.apache.doris.nereids.rules.analysis.BindRelation;
import org.apache.doris.nereids.rules.analysis.BindSlotReference;
+import org.apache.doris.nereids.rules.analysis.FillUpMissingSlots;
import org.apache.doris.nereids.rules.analysis.ProjectToGlobalAggregate;
import org.apache.doris.nereids.rules.analysis.RegisterCTE;
-import org.apache.doris.nereids.rules.analysis.ResolveAggregateFunctions;
import org.apache.doris.nereids.rules.analysis.Scope;
import org.apache.doris.nereids.rules.analysis.UserAuthentication;
@@ -55,7 +55,7 @@ public class AnalyzeRulesJob extends BatchRulesJob {
new ProjectToGlobalAggregate()
)),
topDownBatch(ImmutableList.of(
- new ResolveAggregateFunctions()
+ new FillUpMissingSlots()
))
));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
index f6bf96856d..b0f822cac1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
@@ -52,8 +52,8 @@ public class PlanPostProcessors {
Builder<PlanPostProcessor> builder = ImmutableList.builder();
if (cascadesContext.getConnectContext().getSessionVariable().isEnableNereidsRuntimeFilter()) {
builder.add(new RuntimeFilterGenerator());
- builder.add(new Validator());
}
+ builder.add(new Validator());
return builder.build();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index b094375c82..bbe81e49c9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -46,9 +46,10 @@ public enum RuleType {
BINDING_FILTER_FUNCTION(RuleTypeClass.REWRITE),
BINDING_HAVING_FUNCTION(RuleTypeClass.REWRITE),
BINDING_SORT_FUNCTION(RuleTypeClass.REWRITE),
- RESOLVE_HAVING_AGGREGATE_FUNCTIONS(RuleTypeClass.REWRITE),
- RESOLVE_SORT_AGGREGATE_FUNCTIONS(RuleTypeClass.REWRITE),
- RESOLVE_SORT_HAVING_AGGREGATE_FUNCTIONS(RuleTypeClass.REWRITE),
+ FILL_UP_HAVING_AGGREGATE_FUNCTIONS(RuleTypeClass.REWRITE),
+ FILL_UP_SORT_AGGREGATE_FUNCTIONS(RuleTypeClass.REWRITE),
+ FILL_UP_SORT_HAVING_AGGREGATE_FUNCTIONS(RuleTypeClass.REWRITE),
+ FILL_UP_SORT_PROJECT(RuleTypeClass.REWRITE),
RESOLVE_PROJECT_ALIAS(RuleTypeClass.REWRITE),
RESOLVE_AGGREGATE_ALIAS(RuleTypeClass.REWRITE),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
index 07b1763afa..59cacf524b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSlotReference.java
@@ -160,12 +160,15 @@ public class BindSlotReference implements AnalysisRuleFactory {
})
),
RuleType.BINDING_SORT_SLOT.build(
- logicalSort().when(Plan::canBind).thenApply(ctx -> {
- LogicalSort<GroupPlan> sort = ctx.root;
+ logicalSort(logicalProject()).when(Plan::canBind).thenApply(ctx -> {
+ LogicalSort<LogicalProject<GroupPlan>> sort = ctx.root;
List<OrderKey> sortItemList = sort.getOrderKeys()
.stream()
.map(orderKey -> {
Expression item = bind(orderKey.getExpr(), sort.children(), sort, ctx.cascadesContext);
+ if (item.containsType(UnboundSlot.class)) {
+ item = bind(item, sort.child().children(), sort, ctx.cascadesContext);
+ }
return new OrderKey(item, orderKey.isAsc(), orderKey.isNullFirst());
}).collect(Collectors.toList());
@@ -291,7 +294,9 @@ public class BindSlotReference implements AnalysisRuleFactory {
List<Slot> bounded = boundedOpt.get();
switch (bounded.size()) {
case 0:
- throw new AnalysisException(String.format("Cannot find column %s.", unboundSlot.toSql()));
+ // just return, give a chance to bind on another slot.
+ // if unbound finally, check will throw exception
+ return unboundSlot;
case 1:
if (!foundInThisScope) {
getScope().getOuterScope().get().getCorrelatedSlots().add(bounded.get(0));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java
index 4064294dd8..9d50db7c36 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.rules.analysis;
+import org.apache.doris.nereids.analyzer.UnboundSlot;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.rules.RuleType;
@@ -24,7 +25,11 @@ import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.typecoercion.TypeCheckResult;
import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.commons.lang.StringUtils;
+
import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
/**
* Check analysis rule to check semantic correct after analysis by Nereids.
@@ -32,7 +37,7 @@ import java.util.Optional;
public class CheckAnalysis extends OneAnalysisRuleFactory {
@Override
public Rule build() {
- return any().then(this::checkExpressionInputTypes).toRule(RuleType.CHECK_ANALYSIS);
+ return any().then(plan -> checkExpressionInputTypes(checkBound(plan))).toRule(RuleType.CHECK_ANALYSIS);
}
private Plan checkExpressionInputTypes(Plan plan) {
@@ -46,4 +51,18 @@ public class CheckAnalysis extends OneAnalysisRuleFactory {
}
return plan;
}
+
+ private Plan checkBound(Plan plan) {
+ Set<UnboundSlot> unboundSlots = plan.getExpressions().stream()
+ .<Set<UnboundSlot>>map(e -> e.collect(UnboundSlot.class::isInstance))
+ .flatMap(Set::stream)
+ .collect(Collectors.toSet());
+ if (!unboundSlots.isEmpty()) {
+ throw new AnalysisException(String.format("Cannot find column %s.",
+ StringUtils.join(unboundSlots.stream()
+ .map(UnboundSlot::toSql)
+ .collect(Collectors.toSet()), ", ")));
+ }
+ return plan;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ResolveAggregateFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FillUpMissingSlots.java
similarity index 86%
rename from fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ResolveAggregateFunctions.java
rename to fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FillUpMissingSlots.java
index 7034b489bd..9215363e95 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ResolveAggregateFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FillUpMissingSlots.java
@@ -37,6 +37,7 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
import org.apache.doris.nereids.util.ExpressionUtils;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableList.Builder;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Streams;
@@ -44,16 +45,17 @@ import com.google.common.collect.Streams;
import java.util.List;
import java.util.Map;
import java.util.Optional;
+import java.util.Set;
import java.util.stream.Collectors;
/**
* Resolve having clause to the aggregation.
*/
-public class ResolveAggregateFunctions implements AnalysisRuleFactory {
+public class FillUpMissingSlots implements AnalysisRuleFactory {
@Override
public List<Rule> buildRules() {
return ImmutableList.of(
- RuleType.RESOLVE_SORT_AGGREGATE_FUNCTIONS.build(
+ RuleType.FILL_UP_SORT_AGGREGATE_FUNCTIONS.build(
logicalSort(logicalAggregate())
.when(sort -> sort.getExpressions().stream()
.anyMatch(e -> e.containsType(AggregateFunction.class)))
@@ -72,7 +74,7 @@ public class ResolveAggregateFunctions implements AnalysisRuleFactory {
});
})
),
- RuleType.RESOLVE_SORT_HAVING_AGGREGATE_FUNCTIONS.build(
+ RuleType.FILL_UP_SORT_HAVING_AGGREGATE_FUNCTIONS.build(
logicalSort(logicalHaving(logicalAggregate()))
.when(sort -> sort.getExpressions().stream()
.anyMatch(e -> e.containsType(AggregateFunction.class)))
@@ -91,7 +93,7 @@ public class ResolveAggregateFunctions implements AnalysisRuleFactory {
});
})
),
- RuleType.RESOLVE_HAVING_AGGREGATE_FUNCTIONS.build(
+ RuleType.FILL_UP_HAVING_AGGREGATE_FUNCTIONS.build(
logicalHaving(logicalAggregate()).then(having -> {
LogicalAggregate<GroupPlan> aggregate = having.child();
Resolver resolver = new Resolver(aggregate);
@@ -102,6 +104,27 @@ public class ResolveAggregateFunctions implements AnalysisRuleFactory {
return new LogicalFilter<>(newPredicates, a);
});
})
+ ),
+ RuleType.FILL_UP_SORT_PROJECT.build(
+ logicalSort(logicalProject())
+ .when(sort -> sort.getExpressions().stream()
+ .map(Expression::getInputSlots)
+ .flatMap(Set::stream)
+ .anyMatch(s -> !sort.child().getOutputSet().contains(s)))
+ .then(sort -> {
+ final Builder<NamedExpression> projectionsBuilder = ImmutableList.builder();
+ projectionsBuilder.addAll(sort.child().getProjects());
+ Set<Slot> notExistedInProject = sort.getExpressions().stream()
+ .map(Expression::getInputSlots)
+ .flatMap(Set::stream)
+ .filter(s -> !sort.child().getOutputSet().contains(s))
+ .collect(Collectors.toSet());
+ projectionsBuilder.addAll(notExistedInProject);
+ return new LogicalProject(sort.child().getOutput(),
+ new LogicalSort<>(sort.getOrderKeys(),
+ new LogicalProject<>(projectionsBuilder.build(),
+ sort.child().child())));
+ })
)
);
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
index 65a18b8591..8a3ead9f75 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/clickbench/AnalyzeClickBenchTest.java
@@ -140,20 +140,20 @@ public class AnalyzeClickBenchTest extends ClickBenchTestBase {
checkAnalyze(ClickBenchUtils.Q23);
}
- // @Test
- // public void q24() {
- // checkAnalyze(ClickBenchUtils.Q24);
- // }
+ @Test
+ public void q24() {
+ checkAnalyze(ClickBenchUtils.Q24);
+ }
@Test
public void q25() {
checkAnalyze(ClickBenchUtils.Q25);
}
- // @Test
- // public void q26() {
- // checkAnalyze(ClickBenchUtils.Q26);
- // }
+ @Test
+ public void q26() {
+ checkAnalyze(ClickBenchUtils.Q26);
+ }
@Test
public void q27() {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/AnalyzeTPCHTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/AnalyzeTPCHTest.java
index ae0dfc754e..7be4a60328 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/AnalyzeTPCHTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/AnalyzeTPCHTest.java
@@ -182,11 +182,10 @@ public class AnalyzeTPCHTest extends TPCHTestBase {
checkAnalyze(TPCHUtils.Q21_rewrite);
}
- // NOTE: not support '1 for 2' syntax
- /*@Test
+ @Test
public void q22() {
checkAnalyze(TPCHUtils.Q22);
- }*/
+ }
@Test
public void q22_rewrite() {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/TPCHUtils.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/TPCHUtils.java
index ee8c4b25e4..61be0f086e 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/TPCHUtils.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/datasets/tpch/TPCHUtils.java
@@ -21,7 +21,7 @@ import org.apache.doris.utframe.TestWithFeService;
public class TPCHUtils {
- public static final String Q1 = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=false, enable_projection=false) */\n"
+ public static final String Q1 = "select\n"
+ " l_returnflag,\n"
+ " l_linestatus,\n"
+ " sum(l_quantity) as sum_qty,\n"
@@ -88,7 +88,7 @@ public class TPCHUtils {
+ " p_partkey\n"
+ "limit 100;";
- public static final String Q2_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=1, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */\n"
+ public static final String Q2_rewrite = "select\n"
+ " s_acctbal,\n"
+ " s_name,\n"
+ " n_name,\n"
@@ -152,7 +152,7 @@ public class TPCHUtils {
+ " o_orderdate\n"
+ "limit 10;";
- public static String Q3_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */\n"
+ public static String Q3_rewrite = "select \n"
+ " l_orderkey,\n"
+ " sum(l_extendedprice * (1 - l_discount)) as revenue,\n"
+ " o_orderdate,\n"
@@ -198,7 +198,7 @@ public class TPCHUtils {
+ "order by\n"
+ " o_orderpriority;";
- public static String Q4_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=1, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=false, enable_projection=true) */\n"
+ public static String Q4_rewrite = "select \n"
+ " o_orderpriority,\n"
+ " count(*) as order_count\n"
+ "from\n"
@@ -398,7 +398,7 @@ public class TPCHUtils {
+ " revenue desc\n"
+ "limit 20;";
- public static final String Q11 = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q11 = "select \n"
+ " ps_partkey,\n"
+ " sum(ps_supplycost * ps_availqty) as value\n"
+ "from\n"
@@ -454,7 +454,7 @@ public class TPCHUtils {
+ "order by\n"
+ " l_shipmode;";
- public static final String Q12_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q12_rewrite = "select \n"
+ " l_shipmode,\n"
+ " sum(case\n"
+ " when o_orderpriority = '1-URGENT'\n"
@@ -518,7 +518,7 @@ public class TPCHUtils {
+ " and l_shipdate >= date '1995-09-01'\n"
+ " and l_shipdate < date '1995-10-01';";
- public static final String Q14_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q14_rewrite = "select \n"
+ " 100.00 * sum(case\n"
+ " when p_type like 'PROMO%'\n"
+ " then l_extendedprice * (1 - l_discount)\n"
@@ -563,7 +563,7 @@ public class TPCHUtils {
+ "order by\n"
+ "\ts_suppkey;";
- public static final String Q15_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=4, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q15_rewrite = "select \n"
+ " s_suppkey,\n"
+ " s_name,\n"
+ " s_address,\n"
@@ -632,7 +632,7 @@ public class TPCHUtils {
+ " l_partkey = p_partkey\n"
+ " );";
- public static final String Q17_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=1, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q17_rewrite = "select \n"
+ " sum(l_extendedprice) / 7.0 as avg_yearly\n"
+ "from\n"
+ " lineitem join [broadcast]\n"
@@ -686,7 +686,7 @@ public class TPCHUtils {
+ " o_orderdate\n"
+ "limit 100;";
- public static final String Q18_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q18_rewrite = "select \n"
+ " c_name,\n"
+ " c_custkey,\n"
+ " t3.o_orderkey,\n"
@@ -799,7 +799,7 @@ public class TPCHUtils {
+ "order by\n"
+ " s_name;";
- public static final String Q20_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q20_rewrite = "select \n"
+ "s_name, s_address from\n"
+ "supplier left semi join\n"
+ "(\n"
@@ -866,7 +866,7 @@ public class TPCHUtils {
+ " s_name\n"
+ "limit 100;";
- public static final String Q21_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=true, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q21_rewrite = "select \n"
+ "s_name, count(*) as numwait\n"
+ "from orders join\n"
+ "(\n"
@@ -905,12 +905,12 @@ public class TPCHUtils {
+ "from\n"
+ " (\n"
+ " select\n"
- + " substring(c_phone from 1 for 2) as cntrycode,\n"
+ + " substring(c_phone, 1, 2) as cntrycode,\n"
+ " c_acctbal\n"
+ " from\n"
+ " customer\n"
+ " where\n"
- + " substring(c_phone from 1 for 2) in\n"
+ + " substring(c_phone, 1, 2) in\n"
+ " ('13', '31', '23', '29', '30', '18', '17')\n"
+ " and c_acctbal > (\n"
+ " select\n"
@@ -919,7 +919,7 @@ public class TPCHUtils {
+ " customer\n"
+ " where\n"
+ " c_acctbal > 0.00\n"
- + " and substring(c_phone from 1 for 2) in\n"
+ + " and substring(c_phone, 1, 2) in\n"
+ " ('13', '31', '23', '29', '30', '18', '17')\n"
+ " )\n"
+ " and not exists (\n"
@@ -936,7 +936,7 @@ public class TPCHUtils {
+ "order by\n"
+ " cntrycode;";
- public static final String Q22_rewrite = "select /*+SET_VAR(exec_mem_limit=8589934592, parallel_fragment_exec_instance_num=8, enable_vectorized_engine=true, batch_size=4096, disable_join_reorder=false, enable_cost_based_join_reorder=true, enable_projection=true) */\n"
+ public static final String Q22_rewrite = "select \n"
+ " cntrycode,\n"
+ " count(*) as numcust,\n"
+ " sum(c_acctbal) as totacctbal\n"
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ResolveAggregateFunctionsTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/FillUpMissingSlotsTest.java
similarity index 99%
rename from fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ResolveAggregateFunctionsTest.java
rename to fe/fe-core/src/test/java/org/apache/doris/nereids/parser/FillUpMissingSlotsTest.java
index 10bbe7ae6f..c63310e518 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ResolveAggregateFunctionsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/FillUpMissingSlotsTest.java
@@ -48,7 +48,7 @@ import org.junit.jupiter.api.Test;
import java.util.stream.Collectors;
-public class ResolveAggregateFunctionsTest extends AnalyzeCheckTestBase implements PatternMatchSupported {
+public class FillUpMissingSlotsTest extends AnalyzeCheckTestBase implements PatternMatchSupported {
@Override
public void runBeforeAll() throws Exception {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/LogicalPlanBuilder.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/LogicalPlanBuilder.java
index 2535a4ac41..fb9f9a7b49 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/LogicalPlanBuilder.java
@@ -61,11 +61,6 @@ public class LogicalPlanBuilder {
return from(scan);
}
- public LogicalPlanBuilder projectWithExprs(List<NamedExpression> projectExprs) {
- LogicalProject<LogicalPlan> project = new LogicalProject<>(projectExprs, this.plan);
- return from(project);
- }
-
public LogicalPlanBuilder project(List<Integer> slotsIndex) {
List<NamedExpression> projectExprs = Lists.newArrayList();
for (Integer index : slotsIndex) {
@@ -143,26 +138,26 @@ public class LogicalPlanBuilder {
for (Integer index : outputExprsIndex) {
outputBuilder.add(this.plan.getOutput().get(index));
}
- ImmutableList<NamedExpression> outputExpresList = outputBuilder.build();
+ ImmutableList<NamedExpression> outputExprsList = outputBuilder.build();
- LogicalAggregate<Plan> agg = new LogicalAggregate<>(groupByKeys, outputExpresList, this.plan);
+ LogicalAggregate<Plan> agg = new LogicalAggregate<>(groupByKeys, outputExprsList, this.plan);
return from(agg);
}
public LogicalPlanBuilder aggGroupUsingIndex(List<Integer> groupByKeysIndex,
- List<NamedExpression> outputExpresList) {
+ List<NamedExpression> outputExprsList) {
Builder<Expression> groupByBuilder = ImmutableList.builder();
for (Integer index : groupByKeysIndex) {
groupByBuilder.add(this.plan.getOutput().get(index));
}
ImmutableList<Expression> groupByKeys = groupByBuilder.build();
- LogicalAggregate<Plan> agg = new LogicalAggregate<>(groupByKeys, outputExpresList, this.plan);
+ LogicalAggregate<Plan> agg = new LogicalAggregate<>(groupByKeys, outputExprsList, this.plan);
return from(agg);
}
- public LogicalPlanBuilder agg(List<Expression> groupByKeys, List<NamedExpression> outputExpresList) {
- LogicalAggregate<Plan> agg = new LogicalAggregate<>(groupByKeys, outputExpresList, this.plan);
+ public LogicalPlanBuilder agg(List<Expression> groupByKeys, List<NamedExpression> outputExprsList) {
+ LogicalAggregate<Plan> agg = new LogicalAggregate<>(groupByKeys, outputExprsList, this.plan);
return from(agg);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 08/39: [Bug](Bitmap) fix sub_bitmap calculate wrong result to return null (#13978)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 682dc378ddfb7a4f5669973d43979743ea2cc634
Author: zhangstar333 <87...@users.noreply.github.com>
AuthorDate: Tue Nov 8 14:10:12 2022 +0800
[Bug](Bitmap) fix sub_bitmap calculate wrong result to return null (#13978)
fix sub_bitmap calculate wrong result to return null
---
be/src/util/bitmap_value.h | 89 +++++++++++++++++-----
.../docs/ecosystem/external-table/jdbc-of-doris.md | 2 +-
.../docs/ecosystem/external-table/jdbc-of-doris.md | 2 +-
.../bitmap_functions/test_bitmap_function.out | 12 +++
.../bitmap_functions/test_bitmap_function.groovy | 7 +-
5 files changed, 88 insertions(+), 24 deletions(-)
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index 2cbc0c00f9..e8f10a4f39 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -1682,19 +1682,35 @@ public:
*/
int64_t sub_range(const int64_t& range_start, const int64_t& range_end,
BitmapValue* ret_bitmap) {
- int64_t count = 0;
- for (auto it = _bitmap.begin(); it != _bitmap.end(); ++it) {
- if (*it < range_start) {
- continue;
- }
- if (*it < range_end) {
- ret_bitmap->add(*it);
- ++count;
+ switch (_type) {
+ case EMPTY:
+ return 0;
+ case SINGLE: {
+ //only single value, so _sv must in [range_start,range_end)
+ if (range_start <= _sv && _sv < range_end) {
+ ret_bitmap->add(_sv);
+ return 1;
} else {
- break;
+ return 0;
}
}
- return count;
+ case BITMAP: {
+ int64_t count = 0;
+ for (auto it = _bitmap.begin(); it != _bitmap.end(); ++it) {
+ if (*it < range_start) {
+ continue;
+ }
+ if (*it < range_end) {
+ ret_bitmap->add(*it);
+ ++count;
+ } else {
+ break;
+ }
+ }
+ return count;
+ }
+ }
+ return 0;
}
/**
@@ -1705,19 +1721,35 @@ public:
*/
int64_t sub_limit(const int64_t& range_start, const int64_t& cardinality_limit,
BitmapValue* ret_bitmap) {
- int64_t count = 0;
- for (auto it = _bitmap.begin(); it != _bitmap.end(); ++it) {
- if (*it < range_start) {
- continue;
- }
- if (count < cardinality_limit) {
- ret_bitmap->add(*it);
- ++count;
+ switch (_type) {
+ case EMPTY:
+ return 0;
+ case SINGLE: {
+ //only single value, so range_start must less than _sv
+ if (range_start > _sv) {
+ return 0;
} else {
- break;
+ ret_bitmap->add(_sv);
+ return 1;
}
}
- return count;
+ case BITMAP: {
+ int64_t count = 0;
+ for (auto it = _bitmap.begin(); it != _bitmap.end(); ++it) {
+ if (*it < range_start) {
+ continue;
+ }
+ if (count < cardinality_limit) {
+ ret_bitmap->add(*it);
+ ++count;
+ } else {
+ break;
+ }
+ }
+ return count;
+ }
+ }
+ return 0;
}
/**
@@ -1726,8 +1758,23 @@ public:
* Analog of the substring string function, but for bitmap.
*/
int64_t offset_limit(const int64_t& offset, const int64_t& limit, BitmapValue* ret_bitmap) {
- if (std::abs(offset) >= _bitmap.cardinality()) {
+ switch (_type) {
+ case EMPTY:
return 0;
+ case SINGLE: {
+ //only single value, so offset must start 0
+ if (offset == 0) {
+ ret_bitmap->add(_sv);
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ case BITMAP: {
+ if (std::abs(offset) >= _bitmap.cardinality()) {
+ return 0;
+ }
+ }
}
int64_t abs_offset = offset;
if (offset < 0) {
diff --git a/docs/en/docs/ecosystem/external-table/jdbc-of-doris.md b/docs/en/docs/ecosystem/external-table/jdbc-of-doris.md
index 1eb5ce09a0..029e358957 100644
--- a/docs/en/docs/ecosystem/external-table/jdbc-of-doris.md
+++ b/docs/en/docs/ecosystem/external-table/jdbc-of-doris.md
@@ -75,7 +75,7 @@ Parameter Description:
| **password** | Password information corresponding to the user。|
| **jdbc_url** | The URL protocol of JDBC, including database type, IP address, port number and database name, has different formats for different database protocols. for example mysql: "jdbc:mysql://127.0.0.1:3306/test"。|
| **driver_class** | The class name of the driver package for accessing the external database,for example mysql:com.mysql.jdbc.Driver. |
-| **driver_url** | The package driver URL used to download and access external databases。http://IP:port/mysql-connector-java-5.1.47.jar .|
+| **driver_url** | The package driver URL used to download and access external databases。http://IP:port/mysql-connector-java-5.1.47.jar . During the local test of one BE, the jar package can be placed in the local path, "driver_url"=“ file:///home/disk1/pathTo/mysql-connector-java-5.1.47.jar ", In case of multiple BE test, Must ensure that they have the same path information|
| **resource** | The resource name that depends on when creating the external table in Doris corresponds to the name when creating the resource in the previous step.|
| **table** | The table name mapped to the external database when creating the external table in Doris.|
| **table_type** | When creating an appearance in Doris, the table comes from that database. for example mysql,postgresql,sqlserver,oracle.|
diff --git a/docs/zh-CN/docs/ecosystem/external-table/jdbc-of-doris.md b/docs/zh-CN/docs/ecosystem/external-table/jdbc-of-doris.md
index 461fa5838b..3d8f8daf66 100644
--- a/docs/zh-CN/docs/ecosystem/external-table/jdbc-of-doris.md
+++ b/docs/zh-CN/docs/ecosystem/external-table/jdbc-of-doris.md
@@ -72,7 +72,7 @@ PROPERTIES (
| **password** | 该用户对应的密码信息 |
| **jdbc_url** | JDBC的URL协议,包括数据库类型,IP地址,端口号和数据库名,不同数据库协议格式不一样。例如mysql: "jdbc:mysql://127.0.0.1:3306/test"。|
| **driver_class** | 访问外表数据库的驱动包类名,例如mysql是:com.mysql.jdbc.Driver. |
-| **driver_url** | 用于下载访问外部数据库的jar包驱动URL。http://IP:port/mysql-connector-java-5.1.47.jar |
+| **driver_url** | 用于下载访问外部数据库的jar包驱动URL。http://IP:port/mysql-connector-java-5.1.47.jar。本地单机测试时,可将jar包放在本地路径下,"driver_url"="file:///home/disk1/pathTo/mysql-connector-java-5.1.47.jar",多机时需保证具有完全相同的路径信息。 |
| **resource** | 在Doris中建立外表时依赖的资源名,对应上步创建资源时的名字。|
| **table** | 在Doris中建立外表时,与外部数据库相映射的表名。|
| **table_type** | 在Doris中建立外表时,该表来自那个数据库。例如mysql,postgresql,sqlserver,oracle|
diff --git a/regression-test/data/query_p0/sql_functions/bitmap_functions/test_bitmap_function.out b/regression-test/data/query_p0/sql_functions/bitmap_functions/test_bitmap_function.out
index 74ea0e50ff..ed771ea325 100644
--- a/regression-test/data/query_p0/sql_functions/bitmap_functions/test_bitmap_function.out
+++ b/regression-test/data/query_p0/sql_functions/bitmap_functions/test_bitmap_function.out
@@ -259,3 +259,15 @@ false
-- !sql --
[3, 4, 100, 200]
+-- !sql --
+1,2,3
+
+-- !sql --
+1
+
+-- !sql --
+100
+
+-- !sql --
+20221103
+
diff --git a/regression-test/suites/query_p0/sql_functions/bitmap_functions/test_bitmap_function.groovy b/regression-test/suites/query_p0/sql_functions/bitmap_functions/test_bitmap_function.groovy
index b429b30668..8669f961a4 100644
--- a/regression-test/suites/query_p0/sql_functions/bitmap_functions/test_bitmap_function.groovy
+++ b/regression-test/suites/query_p0/sql_functions/bitmap_functions/test_bitmap_function.groovy
@@ -183,7 +183,12 @@ suite("test_bitmap_function") {
qt_sql """ select orthogonal_bitmap_intersect_count(members, tag_group, 1150000, 1150001, 390006) from ${arthogonalBitmapTable} where tag_group in ( 1150000, 1150001, 390006); """
qt_sql """ select orthogonal_bitmap_union_count(members) from ${arthogonalBitmapTable} where tag_group in ( 1150000, 1150001, 390006); """
- qt_sql """ select bitmap_to_array(user_id) from ${intersectCountTable}; """
+ qt_sql """ select bitmap_to_array(user_id) from ${intersectCountTable} order by dt desc; """
qt_sql """ select bitmap_to_array(bitmap_empty()); """
qt_sql """ select bitmap_to_array(bitmap_from_string('100,200,3,4')); """
+
+ qt_sql """ select bitmap_to_string(sub_bitmap(bitmap_from_string('1,2,3,4,5'), 0, 3)) value; """
+ qt_sql """ select bitmap_to_string(sub_bitmap(bitmap_from_string('1'), 0, 3)) value; """
+ qt_sql """ select bitmap_to_string(bitmap_subset_limit(bitmap_from_string('100'), 0, 3)) value; """
+ qt_sql """ select bitmap_to_string(bitmap_subset_in_range(bitmap_from_string('20221103'), 0, 20221104)) date_list_bitmap; """
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 29/39: [Compile](join) Boost compiling and linking (#14081)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f5ec89b45b5315a540573e473065fbba49bcd2a2
Author: Gabriel <ga...@gmail.com>
AuthorDate: Wed Nov 9 11:27:46 2022 +0800
[Compile](join) Boost compiling and linking (#14081)
---
be/src/vec/exec/join/vhash_join_node.cpp | 264 +++++++++++++++++--------------
be/src/vec/exec/join/vhash_join_node.h | 60 +++----
2 files changed, 163 insertions(+), 161 deletions(-)
diff --git a/be/src/vec/exec/join/vhash_join_node.cpp b/be/src/vec/exec/join/vhash_join_node.cpp
index 0b326db114..8c0d5b3f7a 100644
--- a/be/src/vec/exec/join/vhash_join_node.cpp
+++ b/be/src/vec/exec/join/vhash_join_node.cpp
@@ -58,8 +58,7 @@ struct ProcessHashTableBuild {
_offset(offset),
_build_side_compute_hash_timer(join_node->_build_side_compute_hash_timer) {}
- template <bool need_null_map_for_build, bool ignore_null, bool build_unique,
- bool has_runtime_filter, bool short_circuit_for_null>
+ template <bool ignore_null, bool short_circuit_for_null>
void run(HashTableContext& hash_table_ctx, ConstNullMapPtr null_map, bool* has_null_key) {
using KeyGetter = typename HashTableContext::State;
using Mapped = typename HashTableContext::Mapped;
@@ -76,14 +75,15 @@ struct ProcessHashTableBuild {
SCOPED_TIMER(_join_node->_build_table_insert_timer);
// only not build_unique, we need expanse hash table before insert data
- if constexpr (!build_unique) {
+ if (!_join_node->_build_unique) {
// _rows contains null row, which will cause hash table resize to be large.
hash_table_ctx.hash_table.expanse_for_add_elem(_rows);
}
hash_table_ctx.hash_table.reset_resize_timer();
vector<int>& inserted_rows = _join_node->_inserted_rows[&_acquired_block];
- if constexpr (has_runtime_filter) {
+ bool has_runtime_filter = !_join_node->_runtime_filter_descs.empty();
+ if (has_runtime_filter) {
inserted_rows.reserve(_batch_size);
}
@@ -92,14 +92,14 @@ struct ProcessHashTableBuild {
{
SCOPED_TIMER(_build_side_compute_hash_timer);
for (size_t k = 0; k < _rows; ++k) {
- if constexpr (ignore_null && need_null_map_for_build) {
+ if constexpr (ignore_null) {
if ((*null_map)[k]) {
continue;
}
}
// If apply short circuit strategy for null value (e.g. join operator is
// NULL_AWARE_LEFT_ANTI_JOIN), we build hash table until we meet a null value.
- if constexpr (short_circuit_for_null && need_null_map_for_build) {
+ if constexpr (short_circuit_for_null) {
if ((*null_map)[k]) {
DCHECK(has_null_key);
*has_null_key = true;
@@ -116,37 +116,52 @@ struct ProcessHashTableBuild {
}
}
- for (size_t k = 0; k < _rows; ++k) {
- if constexpr (ignore_null && need_null_map_for_build) {
- if ((*null_map)[k]) {
- continue;
- }
- }
-
- auto emplace_result = key_getter.emplace_key(hash_table_ctx.hash_table,
- _build_side_hash_values[k], k, arena);
- if (k + PREFETCH_STEP < _rows) {
- key_getter.template prefetch_by_hash<false>(
- hash_table_ctx.hash_table, _build_side_hash_values[k + PREFETCH_STEP]);
- }
+ bool build_unique = _join_node->_build_unique;
+#define EMPLACE_IMPL(stmt) \
+ for (size_t k = 0; k < _rows; ++k) { \
+ if constexpr (ignore_null) { \
+ if ((*null_map)[k]) { \
+ continue; \
+ } \
+ } \
+ auto emplace_result = key_getter.emplace_key(hash_table_ctx.hash_table, \
+ _build_side_hash_values[k], k, arena); \
+ if (k + PREFETCH_STEP < _rows) { \
+ key_getter.template prefetch_by_hash<false>( \
+ hash_table_ctx.hash_table, _build_side_hash_values[k + PREFETCH_STEP]); \
+ } \
+ stmt; \
+ }
- if (emplace_result.is_inserted()) {
- new (&emplace_result.get_mapped()) Mapped({k, _offset});
- if constexpr (has_runtime_filter) {
- inserted_rows.push_back(k);
- }
- } else {
- if constexpr (!build_unique) {
- /// The first element of the list is stored in the value of the hash table, the rest in the pool.
- emplace_result.get_mapped().insert({k, _offset}, _join_node->_arena);
- if constexpr (has_runtime_filter) {
+ if (has_runtime_filter && build_unique) {
+ EMPLACE_IMPL(
+ if (emplace_result.is_inserted()) {
+ new (&emplace_result.get_mapped()) Mapped({k, _offset});
inserted_rows.push_back(k);
- }
- } else {
- _skip_rows++;
- }
- }
+ } else { _skip_rows++; });
+ } else if (has_runtime_filter && !build_unique) {
+ EMPLACE_IMPL(
+ if (emplace_result.is_inserted()) {
+ new (&emplace_result.get_mapped()) Mapped({k, _offset});
+ inserted_rows.push_back(k);
+ } else {
+ emplace_result.get_mapped().insert({k, _offset}, _join_node->_arena);
+ inserted_rows.push_back(k);
+ });
+ } else if (!has_runtime_filter && build_unique) {
+ EMPLACE_IMPL(
+ if (emplace_result.is_inserted()) {
+ new (&emplace_result.get_mapped()) Mapped({k, _offset});
+ } else { _skip_rows++; });
+ } else {
+ EMPLACE_IMPL(
+ if (emplace_result.is_inserted()) {
+ new (&emplace_result.get_mapped()) Mapped({k, _offset});
+ } else {
+ emplace_result.get_mapped().insert({k, _offset}, _join_node->_arena);
+ });
}
+#undef EMPLACE_IMPL
COUNTER_UPDATE(_join_node->_build_table_expanse_timer,
hash_table_ctx.hash_table.get_resize_timer_value());
@@ -197,9 +212,8 @@ private:
HashJoinNode* _join_node;
};
-template <class JoinOpType, bool ignore_null>
-ProcessHashTableProbe<JoinOpType, ignore_null>::ProcessHashTableProbe(HashJoinNode* join_node,
- int batch_size)
+template <class JoinOpType>
+ProcessHashTableProbe<JoinOpType>::ProcessHashTableProbe(HashJoinNode* join_node, int batch_size)
: _join_node(join_node),
_batch_size(batch_size),
_build_blocks(join_node->_build_blocks),
@@ -219,9 +233,9 @@ ProcessHashTableProbe<JoinOpType, ignore_null>::ProcessHashTableProbe(HashJoinNo
_build_side_output_timer(join_node->_build_side_output_timer),
_probe_side_output_timer(join_node->_probe_side_output_timer) {}
-template <class JoinOpType, bool ignore_null>
+template <class JoinOpType>
template <bool have_other_join_conjunct>
-void ProcessHashTableProbe<JoinOpType, ignore_null>::build_side_output_column(
+void ProcessHashTableProbe<JoinOpType>::build_side_output_column(
MutableColumns& mcol, int column_offset, int column_length,
const std::vector<bool>& output_slot_flags, int size) {
constexpr auto is_semi_anti_join = JoinOpType::value == TJoinOp::RIGHT_ANTI_JOIN ||
@@ -293,11 +307,11 @@ void ProcessHashTableProbe<JoinOpType, ignore_null>::build_side_output_column(
}
}
-template <class JoinOpType, bool ignore_null>
-template <bool have_other_join_conjunct>
-void ProcessHashTableProbe<JoinOpType, ignore_null>::probe_side_output_column(
+template <class JoinOpType>
+void ProcessHashTableProbe<JoinOpType>::probe_side_output_column(
MutableColumns& mcol, const std::vector<bool>& output_slot_flags, int size,
- int last_probe_index, size_t probe_size, bool all_match_one) {
+ int last_probe_index, size_t probe_size, bool all_match_one,
+ bool have_other_join_conjunct) {
auto& probe_block = _join_node->_probe_block;
for (int i = 0; i < output_slot_flags.size(); ++i) {
if (output_slot_flags[i]) {
@@ -314,18 +328,19 @@ void ProcessHashTableProbe<JoinOpType, ignore_null>::probe_side_output_column(
}
}
- if constexpr (JoinOpType::value == TJoinOp::RIGHT_OUTER_JOIN && !have_other_join_conjunct) {
- _tuple_is_null_left_flags->resize_fill(size, 0);
+ if constexpr (JoinOpType::value == TJoinOp::RIGHT_OUTER_JOIN) {
+ if (!have_other_join_conjunct) {
+ _tuple_is_null_left_flags->resize_fill(size, 0);
+ }
}
}
-template <class JoinOpType, bool ignore_null>
-template <bool need_null_map_for_probe, typename HashTableType>
-Status ProcessHashTableProbe<JoinOpType, ignore_null>::do_process(HashTableType& hash_table_ctx,
- ConstNullMapPtr null_map,
- MutableBlock& mutable_block,
- Block* output_block,
- size_t probe_rows) {
+template <class JoinOpType>
+template <bool need_null_map_for_probe, bool ignore_null, typename HashTableType>
+Status ProcessHashTableProbe<JoinOpType>::do_process(HashTableType& hash_table_ctx,
+ ConstNullMapPtr null_map,
+ MutableBlock& mutable_block,
+ Block* output_block, size_t probe_rows) {
auto& probe_index = _join_node->_probe_index;
auto& probe_raw_ptrs = _join_node->_probe_columns;
if (probe_index == 0 && _items_counts.size() < probe_rows) {
@@ -470,7 +485,8 @@ Status ProcessHashTableProbe<JoinOpType, ignore_null>::do_process(HashTableType&
JoinOpType::value != TJoinOp::RIGHT_ANTI_JOIN) {
SCOPED_TIMER(_probe_side_output_timer);
probe_side_output_column(mcol, _join_node->_left_output_slot_flags, current_offset,
- last_probe_index, probe_index - last_probe_index, all_match_one);
+ last_probe_index, probe_index - last_probe_index, all_match_one,
+ false);
}
output_block->swap(mutable_block.to_block());
@@ -478,9 +494,9 @@ Status ProcessHashTableProbe<JoinOpType, ignore_null>::do_process(HashTableType&
return Status::OK();
}
-template <class JoinOpType, bool ignore_null>
-template <bool need_null_map_for_probe, typename HashTableType>
-Status ProcessHashTableProbe<JoinOpType, ignore_null>::do_process_with_other_join_conjuncts(
+template <class JoinOpType>
+template <bool need_null_map_for_probe, bool ignore_null, typename HashTableType>
+Status ProcessHashTableProbe<JoinOpType>::do_process_with_other_join_conjuncts(
HashTableType& hash_table_ctx, ConstNullMapPtr null_map, MutableBlock& mutable_block,
Block* output_block, size_t probe_rows) {
auto& probe_index = _join_node->_probe_index;
@@ -618,9 +634,9 @@ Status ProcessHashTableProbe<JoinOpType, ignore_null>::do_process_with_other_joi
}
{
SCOPED_TIMER(_probe_side_output_timer);
- probe_side_output_column<true>(mcol, _join_node->_left_output_slot_flags,
- current_offset, last_probe_index,
- probe_index - last_probe_index, all_match_one);
+ probe_side_output_column(mcol, _join_node->_left_output_slot_flags, current_offset,
+ last_probe_index, probe_index - last_probe_index,
+ all_match_one, true);
}
output_block->swap(mutable_block.to_block());
@@ -765,11 +781,12 @@ Status ProcessHashTableProbe<JoinOpType, ignore_null>::do_process_with_other_joi
}
}
-template <class JoinOpType, bool ignore_null>
+template <class JoinOpType>
template <typename HashTableType>
-Status ProcessHashTableProbe<JoinOpType, ignore_null>::process_data_in_hashtable(
- HashTableType& hash_table_ctx, MutableBlock& mutable_block, Block* output_block,
- bool* eos) {
+Status ProcessHashTableProbe<JoinOpType>::process_data_in_hashtable(HashTableType& hash_table_ctx,
+ MutableBlock& mutable_block,
+ Block* output_block,
+ bool* eos) {
using Mapped = typename HashTableType::Mapped;
if constexpr (std::is_same_v<Mapped, RowRefListWithFlag> ||
std::is_same_v<Mapped, RowRefListWithFlags>) {
@@ -926,8 +943,6 @@ Status HashJoinNode::init(const TPlanNode& tnode, RuntimeState* state) {
probe_not_ignore_null[conjuncts_index] =
null_aware ||
(_probe_expr_ctxs.back()->root()->is_nullable() && probe_dispose_null);
- _build_side_ignore_null |= (_join_op != TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN &&
- !_store_null_in_hash_table.back());
conjuncts_index++;
}
for (size_t i = 0; i < _probe_expr_ctxs.size(); ++i) {
@@ -1117,7 +1132,7 @@ Status HashJoinNode::get_next(RuntimeState* state, Block* output_block, bool* eo
// so we have to initialize this flag by the first probe block.
if (!_has_set_need_null_map_for_probe) {
_has_set_need_null_map_for_probe = true;
- _need_null_map_for_probe = _need_null_map<false>(_probe_block, res_col_ids);
+ _need_null_map_for_probe = _need_probe_null_map(_probe_block, res_col_ids);
}
if (_need_null_map_for_probe) {
if (_null_map_column == nullptr) {
@@ -1150,17 +1165,17 @@ Status HashJoinNode::get_next(RuntimeState* state, Block* output_block, bool* eo
if (_probe_index < _probe_block.rows()) {
DCHECK(_has_set_need_null_map_for_probe);
- std::visit(
- [&](auto&& arg, auto&& process_hashtable_ctx, auto have_other_join_conjunct,
- auto need_null_map_for_probe) {
- using HashTableProbeType = std::decay_t<decltype(process_hashtable_ctx)>;
- if constexpr (!std::is_same_v<HashTableProbeType, std::monostate>) {
- using HashTableCtxType = std::decay_t<decltype(arg)>;
- if constexpr (have_other_join_conjunct) {
+ if (_have_other_join_conjunct) {
+ std::visit(
+ [&](auto&& arg, auto&& process_hashtable_ctx, auto need_null_map_for_probe,
+ auto ignore_null) {
+ using HashTableProbeType = std::decay_t<decltype(process_hashtable_ctx)>;
+ if constexpr (!std::is_same_v<HashTableProbeType, std::monostate>) {
+ using HashTableCtxType = std::decay_t<decltype(arg)>;
if constexpr (!std::is_same_v<HashTableCtxType, std::monostate>) {
st = process_hashtable_ctx
.template do_process_with_other_join_conjuncts<
- need_null_map_for_probe>(
+ need_null_map_for_probe, ignore_null>(
arg,
need_null_map_for_probe
? &_null_map_column->get_data()
@@ -1170,25 +1185,37 @@ Status HashJoinNode::get_next(RuntimeState* state, Block* output_block, bool* eo
LOG(FATAL) << "FATAL: uninited hash table";
}
} else {
+ LOG(FATAL) << "FATAL: uninited hash probe";
+ }
+ },
+ _hash_table_variants, _process_hashtable_ctx_variants,
+ make_bool_variant(_need_null_map_for_probe),
+ make_bool_variant(_probe_ignore_null));
+ } else {
+ std::visit(
+ [&](auto&& arg, auto&& process_hashtable_ctx, auto need_null_map_for_probe,
+ auto ignore_null) {
+ using HashTableProbeType = std::decay_t<decltype(process_hashtable_ctx)>;
+ if constexpr (!std::is_same_v<HashTableProbeType, std::monostate>) {
+ using HashTableCtxType = std::decay_t<decltype(arg)>;
if constexpr (!std::is_same_v<HashTableCtxType, std::monostate>) {
- st = process_hashtable_ctx
- .template do_process<need_null_map_for_probe>(
- arg,
- need_null_map_for_probe
- ? &_null_map_column->get_data()
- : nullptr,
- mutable_join_block, &temp_block, probe_rows);
+ st = process_hashtable_ctx.template do_process<
+ need_null_map_for_probe, ignore_null>(
+ arg,
+ need_null_map_for_probe ? &_null_map_column->get_data()
+ : nullptr,
+ mutable_join_block, &temp_block, probe_rows);
} else {
LOG(FATAL) << "FATAL: uninited hash table";
}
+ } else {
+ LOG(FATAL) << "FATAL: uninited hash probe";
}
- } else {
- LOG(FATAL) << "FATAL: uninited hash probe";
- }
- },
- _hash_table_variants, _process_hashtable_ctx_variants,
- make_bool_variant(_have_other_join_conjunct),
- make_bool_variant(_need_null_map_for_probe));
+ },
+ _hash_table_variants, _process_hashtable_ctx_variants,
+ make_bool_variant(_need_null_map_for_probe),
+ make_bool_variant(_probe_ignore_null));
+ }
} else if (_probe_eos) {
if (_is_right_semi_anti || (_is_outer_join && _join_op != TJoinOp::LEFT_OUTER_JOIN)) {
std::visit(
@@ -1426,28 +1453,32 @@ Status HashJoinNode::_do_evaluate(Block& block, std::vector<VExprContext*>& expr
return Status::OK();
}
-template <bool BuildSide>
-bool HashJoinNode::_need_null_map(Block& block, const std::vector<int>& res_col_ids) {
+bool HashJoinNode::_need_probe_null_map(Block& block, const std::vector<int>& res_col_ids) {
DCHECK_EQ(_build_expr_ctxs.size(), _probe_expr_ctxs.size());
for (size_t i = 0; i < _build_expr_ctxs.size(); ++i) {
if (!_is_null_safe_eq_join[i]) {
auto column = block.get_by_position(res_col_ids[i]).column.get();
- if constexpr (BuildSide) {
- if (check_and_get_column<ColumnNullable>(*column)) {
- if (!_store_null_in_hash_table[i]) {
- return true;
- }
- }
- } else {
- if (check_and_get_column<ColumnNullable>(*column)) {
- return true;
- }
+ if (check_and_get_column<ColumnNullable>(*column)) {
+ return true;
}
}
}
return false;
}
+void HashJoinNode::_set_build_ignore_flag(Block& block, const std::vector<int>& res_col_ids) {
+ DCHECK_EQ(_build_expr_ctxs.size(), _probe_expr_ctxs.size());
+ for (size_t i = 0; i < _build_expr_ctxs.size(); ++i) {
+ if (!_is_null_safe_eq_join[i]) {
+ auto column = block.get_by_position(res_col_ids[i]).column.get();
+ if (check_and_get_column<ColumnNullable>(*column)) {
+ _build_side_ignore_null |= (_join_op != TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN &&
+ !_store_null_in_hash_table[i]);
+ }
+ }
+ }
+}
+
Status HashJoinNode::_process_build_block(RuntimeState* state, Block& block, uint8_t offset) {
SCOPED_TIMER(_build_table_timer);
size_t rows = block.rows();
@@ -1468,10 +1499,9 @@ Status HashJoinNode::_process_build_block(RuntimeState* state, Block& block, uin
// so we have to initialize this flag by the first build block.
if (!_has_set_need_null_map_for_build) {
_has_set_need_null_map_for_build = true;
- _need_null_map_for_build =
- _short_circuit_for_null_in_build_side || _need_null_map<true>(block, res_col_ids);
+ _set_build_ignore_flag(block, res_col_ids);
}
- if (_need_null_map_for_build) {
+ if (_short_circuit_for_null_in_build_side || _build_side_ignore_null) {
null_map_val = ColumnUInt8::create();
null_map_val->get_data().assign(rows, (uint8_t)0);
}
@@ -1489,27 +1519,24 @@ Status HashJoinNode::_process_build_block(RuntimeState* state, Block& block, uin
},
_hash_table_variants);
- bool has_runtime_filter = !_runtime_filter_descs.empty();
-
std::visit(
- [&](auto&& arg, auto has_null_value, auto build_unique, auto has_runtime_filter_value,
- auto need_null_map_for_build, auto short_circuit_for_null_in_build_side) {
+ [&](auto&& arg, auto has_null_value, auto short_circuit_for_null_in_build_side) {
using HashTableCtxType = std::decay_t<decltype(arg)>;
if constexpr (!std::is_same_v<HashTableCtxType, std::monostate>) {
ProcessHashTableBuild<HashTableCtxType> hash_table_build_process(
rows, block, raw_ptrs, this, state->batch_size(), offset);
- hash_table_build_process.template run<need_null_map_for_build, has_null_value,
- build_unique, has_runtime_filter_value,
- short_circuit_for_null_in_build_side>(
- arg, need_null_map_for_build ? &null_map_val->get_data() : nullptr,
- &_short_circuit_for_null_in_probe_side);
+ hash_table_build_process
+ .template run<has_null_value, short_circuit_for_null_in_build_side>(
+ arg,
+ has_null_value || short_circuit_for_null_in_build_side
+ ? &null_map_val->get_data()
+ : nullptr,
+ &_short_circuit_for_null_in_probe_side);
} else {
LOG(FATAL) << "FATAL: uninited hash table";
}
},
_hash_table_variants, make_bool_variant(_build_side_ignore_null),
- make_bool_variant(_build_unique), make_bool_variant(has_runtime_filter),
- make_bool_variant(_need_null_map_for_build),
make_bool_variant(_short_circuit_for_null_in_build_side));
return st;
@@ -1640,13 +1667,12 @@ void HashJoinNode::_hash_table_init() {
void HashJoinNode::_process_hashtable_ctx_variants_init(RuntimeState* state) {
std::visit(
- [&](auto&& join_op_variants, auto probe_ignore_null) {
+ [&](auto&& join_op_variants) {
using JoinOpType = std::decay_t<decltype(join_op_variants)>;
- _process_hashtable_ctx_variants
- .emplace<ProcessHashTableProbe<JoinOpType, probe_ignore_null>>(
- this, state->batch_size());
+ _process_hashtable_ctx_variants.emplace<ProcessHashTableProbe<JoinOpType>>(
+ this, state->batch_size());
},
- _join_op_variants, make_bool_variant(_probe_ignore_null));
+ _join_op_variants);
}
std::vector<uint16_t> HashJoinNode::_convert_block_to_null(Block& block) {
diff --git a/be/src/vec/exec/join/vhash_join_node.h b/be/src/vec/exec/join/vhash_join_node.h
index 7d29268b39..977edcb29c 100644
--- a/be/src/vec/exec/join/vhash_join_node.h
+++ b/be/src/vec/exec/join/vhash_join_node.h
@@ -183,7 +183,7 @@ using JoinOpVariants =
class VExprContext;
class HashJoinNode;
-template <class JoinOpType, bool ignore_null>
+template <class JoinOpType>
struct ProcessHashTableProbe {
ProcessHashTableProbe(HashJoinNode* join_node, int batch_size);
@@ -192,22 +192,21 @@ struct ProcessHashTableProbe {
void build_side_output_column(MutableColumns& mcol, int column_offset, int column_length,
const std::vector<bool>& output_slot_flags, int size);
- template <bool have_other_join_conjunct = false>
void probe_side_output_column(MutableColumns& mcol, const std::vector<bool>& output_slot_flags,
int size, int last_probe_index, size_t probe_size,
- bool all_match_one);
+ bool all_match_one, bool have_other_join_conjunct);
// Only process the join with no other join conjunt, because of no other join conjunt
// the output block struct is same with mutable block. we can do more opt on it and simplify
// the logic of probe
// TODO: opt the visited here to reduce the size of hash table
- template <bool need_null_map_for_probe, typename HashTableType>
+ template <bool need_null_map_for_probe, bool ignore_null, typename HashTableType>
Status do_process(HashTableType& hash_table_ctx, ConstNullMapPtr null_map,
MutableBlock& mutable_block, Block* output_block, size_t probe_rows);
// In the presence of other join conjunt, the process of join become more complicated.
// each matching join column need to be processed by other join conjunt. so the sturct of mutable block
// and output block may be different
// The output result is determined by the other join conjunt result and same_to_prev struct
- template <bool need_null_map_for_probe, typename HashTableType>
+ template <bool need_null_map_for_probe, bool ignore_null, typename HashTableType>
Status do_process_with_other_join_conjuncts(HashTableType& hash_table_ctx,
ConstNullMapPtr null_map,
MutableBlock& mutable_block, Block* output_block,
@@ -242,40 +241,17 @@ struct ProcessHashTableProbe {
using HashTableCtxVariants = std::variant<
std::monostate,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::INNER_JOIN>, true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_SEMI_JOIN>, true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_ANTI_JOIN>, true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_OUTER_JOIN>,
- true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::FULL_OUTER_JOIN>,
- true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_OUTER_JOIN>,
- true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::CROSS_JOIN>, true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_SEMI_JOIN>,
- true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_ANTI_JOIN>,
- true>,
- ProcessHashTableProbe<
- std::integral_constant<TJoinOp::type, TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN>, true>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::INNER_JOIN>, false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_SEMI_JOIN>,
- false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_ANTI_JOIN>,
- false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_OUTER_JOIN>,
- false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::FULL_OUTER_JOIN>,
- false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_OUTER_JOIN>,
- false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::CROSS_JOIN>, false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_SEMI_JOIN>,
- false>,
- ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_ANTI_JOIN>,
- false>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::INNER_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_SEMI_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_ANTI_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::LEFT_OUTER_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::FULL_OUTER_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_OUTER_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::CROSS_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_SEMI_JOIN>>,
+ ProcessHashTableProbe<std::integral_constant<TJoinOp::type, TJoinOp::RIGHT_ANTI_JOIN>>,
ProcessHashTableProbe<
- std::integral_constant<TJoinOp::type, TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN>, false>>;
+ std::integral_constant<TJoinOp::type, TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN>>>;
class HashJoinNode : public ::doris::ExecNode {
public:
@@ -353,7 +329,6 @@ private:
ColumnUInt8::MutablePtr _null_map_column;
bool _need_null_map_for_probe = false;
bool _has_set_need_null_map_for_probe = false;
- bool _need_null_map_for_build = false;
bool _has_set_need_null_map_for_build = false;
bool _probe_ignore_null = false;
int _probe_index = -1;
@@ -405,8 +380,9 @@ private:
Status _extract_join_column(Block& block, ColumnUInt8::MutablePtr& null_map,
ColumnRawPtrs& raw_ptrs, const std::vector<int>& res_col_ids);
- template <bool BuildSide>
- bool _need_null_map(Block& block, const std::vector<int>& res_col_ids);
+ bool _need_probe_null_map(Block& block, const std::vector<int>& res_col_ids);
+
+ void _set_build_ignore_flag(Block& block, const std::vector<int>& res_col_ids);
void _hash_table_init();
void _process_hashtable_ctx_variants_init(RuntimeState* state);
@@ -430,7 +406,7 @@ private:
template <class HashTableContext>
friend struct ProcessHashTableBuild;
- template <class JoinOpType, bool ignore_null>
+ template <class JoinOpType>
friend struct ProcessHashTableProbe;
template <class HashTableContext>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/39: [Enhancement](Dictionary-codec) update dict once on same segment (#13936)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f4b4e603490aaeb28635e532a0de51e03965ade3
Author: Pxl <px...@qq.com>
AuthorDate: Tue Nov 8 10:59:35 2022 +0800
[Enhancement](Dictionary-codec) update dict once on same segment (#13936)
update dict once on same segment
---
be/src/olap/in_list_predicate.h | 17 ++++++++++++++---
be/src/olap/rowset/beta_rowset.cpp | 9 ++++++---
be/src/olap/rowset/segment_v2/segment.cpp | 11 +++++++----
be/src/olap/rowset/segment_v2/segment.h | 7 +++++--
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 ++
be/src/vec/columns/column.h | 6 ++++++
be/src/vec/columns/column_dictionary.h | 9 +++++++++
be/test/io/cache/remote_file_cache_test.cpp | 2 +-
be/test/olap/rowset/segment_v2/segment_test.cpp | 4 ++--
be/test/tools/benchmark_tool.cpp | 2 +-
10 files changed, 53 insertions(+), 16 deletions(-)
diff --git a/be/src/olap/in_list_predicate.h b/be/src/olap/in_list_predicate.h
index 32b0dc2fb0..b85deeb39c 100644
--- a/be/src/olap/in_list_predicate.h
+++ b/be/src/olap/in_list_predicate.h
@@ -20,11 +20,13 @@
#include <parallel_hashmap/phmap.h>
#include <stdint.h>
+#include <cstdint>
#include <roaring/roaring.hh>
#include <type_traits>
#include "decimal12.h"
#include "olap/column_predicate.h"
+#include "olap/olap_common.h"
#include "olap/rowset/segment_v2/bloom_filter.h"
#include "olap/wrapper_field.h"
#include "runtime/define_primitive_type.h"
@@ -400,7 +402,11 @@ private:
auto* nested_col_ptr = vectorized::check_and_get_column<
vectorized::ColumnDictionary<vectorized::Int32>>(column);
auto& data_array = nested_col_ptr->get_data();
- nested_col_ptr->find_codes(_values, _value_in_dict_flags);
+ auto& _value_in_dict_flags =
+ _segment_id_to_value_in_dict_flags[column->get_rowset_segment_id()];
+ if (_value_in_dict_flags.empty()) {
+ nested_col_ptr->find_codes(_values, _value_in_dict_flags);
+ }
for (uint16_t i = 0; i < size; i++) {
uint16_t idx = sel[i];
@@ -469,7 +475,11 @@ private:
auto* nested_col_ptr = vectorized::check_and_get_column<
vectorized::ColumnDictionary<vectorized::Int32>>(column);
auto& data_array = nested_col_ptr->get_data();
- nested_col_ptr->find_codes(_values, _value_in_dict_flags);
+ auto& _value_in_dict_flags =
+ _segment_id_to_value_in_dict_flags[column->get_rowset_segment_id()];
+ if (_value_in_dict_flags.empty()) {
+ nested_col_ptr->find_codes(_values, _value_in_dict_flags);
+ }
for (uint16_t i = 0; i < size; i++) {
if (is_and ^ flags[i]) {
@@ -543,7 +553,8 @@ private:
}
phmap::flat_hash_set<T> _values;
- mutable std::vector<vectorized::UInt8> _value_in_dict_flags;
+ mutable std::map<std::pair<RowsetId, uint32_t>, std::vector<vectorized::UInt8>>
+ _segment_id_to_value_in_dict_flags;
T _min_value;
T _max_value;
static constexpr PrimitiveType EvalType = (Type == TYPE_CHAR ? TYPE_STRING : Type);
diff --git a/be/src/olap/rowset/beta_rowset.cpp b/be/src/olap/rowset/beta_rowset.cpp
index fdb7b90496..8ed48fdf70 100644
--- a/be/src/olap/rowset/beta_rowset.cpp
+++ b/be/src/olap/rowset/beta_rowset.cpp
@@ -118,7 +118,8 @@ Status BetaRowset::load_segments(std::vector<segment_v2::SegmentSharedPtr>* segm
auto seg_path = segment_file_path(seg_id);
auto cache_path = segment_cache_path(seg_id);
std::shared_ptr<segment_v2::Segment> segment;
- auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, _schema, &segment);
+ auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, rowset_id(), _schema,
+ &segment);
if (!s.ok()) {
LOG(WARNING) << "failed to open segment. " << seg_path << " under rowset "
<< unique_id() << " : " << s.to_string();
@@ -137,7 +138,8 @@ Status BetaRowset::load_segment(int64_t seg_id, segment_v2::SegmentSharedPtr* se
}
auto seg_path = segment_file_path(seg_id);
auto cache_path = segment_cache_path(seg_id);
- auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, _schema, segment);
+ auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, rowset_id(), _schema,
+ segment);
if (!s.ok()) {
LOG(WARNING) << "failed to open segment. " << seg_path << " under rowset " << unique_id()
<< " : " << s.to_string();
@@ -304,7 +306,8 @@ bool BetaRowset::check_current_rowset_segment() {
auto seg_path = segment_file_path(seg_id);
auto cache_path = segment_cache_path(seg_id);
std::shared_ptr<segment_v2::Segment> segment;
- auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, _schema, &segment);
+ auto s = segment_v2::Segment::open(fs, seg_path, cache_path, seg_id, rowset_id(), _schema,
+ &segment);
if (!s.ok()) {
LOG(WARNING) << "segment can not be opened. file=" << seg_path;
return false;
diff --git a/be/src/olap/rowset/segment_v2/segment.cpp b/be/src/olap/rowset/segment_v2/segment.cpp
index 056665d295..baec276993 100644
--- a/be/src/olap/rowset/segment_v2/segment.cpp
+++ b/be/src/olap/rowset/segment_v2/segment.cpp
@@ -43,9 +43,9 @@ namespace segment_v2 {
using io::FileCacheManager;
Status Segment::open(io::FileSystemSPtr fs, const std::string& path, const std::string& cache_path,
- uint32_t segment_id, TabletSchemaSPtr tablet_schema,
+ uint32_t segment_id, RowsetId rowset_id, TabletSchemaSPtr tablet_schema,
std::shared_ptr<Segment>* output) {
- std::shared_ptr<Segment> segment(new Segment(segment_id, tablet_schema));
+ std::shared_ptr<Segment> segment(new Segment(segment_id, rowset_id, tablet_schema));
io::FileReaderSPtr file_reader;
#ifndef BE_TEST
RETURN_IF_ERROR(fs->open_file(path, &file_reader));
@@ -71,8 +71,11 @@ Status Segment::open(io::FileSystemSPtr fs, const std::string& path, const std::
return Status::OK();
}
-Segment::Segment(uint32_t segment_id, TabletSchemaSPtr tablet_schema)
- : _segment_id(segment_id), _tablet_schema(tablet_schema), _meta_mem_usage(0) {}
+Segment::Segment(uint32_t segment_id, RowsetId rowset_id, TabletSchemaSPtr tablet_schema)
+ : _segment_id(segment_id),
+ _rowset_id(rowset_id),
+ _tablet_schema(tablet_schema),
+ _meta_mem_usage(0) {}
Segment::~Segment() {
#ifndef BE_TEST
diff --git a/be/src/olap/rowset/segment_v2/segment.h b/be/src/olap/rowset/segment_v2/segment.h
index a3dcd8c6c6..96fbb60335 100644
--- a/be/src/olap/rowset/segment_v2/segment.h
+++ b/be/src/olap/rowset/segment_v2/segment.h
@@ -62,7 +62,7 @@ using SegmentSharedPtr = std::shared_ptr<Segment>;
class Segment : public std::enable_shared_from_this<Segment> {
public:
static Status open(io::FileSystemSPtr fs, const std::string& path,
- const std::string& cache_path, uint32_t segment_id,
+ const std::string& cache_path, uint32_t segment_id, RowsetId rowset_id,
TabletSchemaSPtr tablet_schema, std::shared_ptr<Segment>* output);
~Segment();
@@ -72,6 +72,8 @@ public:
uint64_t id() const { return _segment_id; }
+ RowsetId rowset_id() const { return _rowset_id; }
+
uint32_t num_rows() const { return _footer.num_rows(); }
Status new_column_iterator(const TabletColumn& tablet_column, ColumnIterator** iter);
@@ -108,7 +110,7 @@ public:
private:
DISALLOW_COPY_AND_ASSIGN(Segment);
- Segment(uint32_t segment_id, TabletSchemaSPtr tablet_schema);
+ Segment(uint32_t segment_id, RowsetId rowset_id, TabletSchemaSPtr tablet_schema);
// open segment file and read the minimum amount of necessary information (footer)
Status _open();
Status _parse_footer();
@@ -120,6 +122,7 @@ private:
io::FileReaderSPtr _file_reader;
uint32_t _segment_id;
+ RowsetId _rowset_id;
TabletSchemaSPtr _tablet_schema;
int64_t _meta_mem_usage;
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 1f8e24a2b2..a85c0e35db 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -1229,6 +1229,8 @@ void SegmentIterator::_convert_dict_code_for_predicate_if_necessary_impl(
ColumnPredicate* predicate) {
auto& column = _current_return_columns[predicate->column_id()];
auto* col_ptr = column.get();
+ column->set_rowset_segment_id({_segment->rowset_id(), _segment->id()});
+
if (PredicateTypeTraits::is_range(predicate->type())) {
col_ptr->convert_dict_codes_if_necessary();
} else if (PredicateTypeTraits::is_bloom_filter(predicate->type())) {
diff --git a/be/src/vec/columns/column.h b/be/src/vec/columns/column.h
index 4b89a002af..c5eeb18d8f 100644
--- a/be/src/vec/columns/column.h
+++ b/be/src/vec/columns/column.h
@@ -20,6 +20,7 @@
#pragma once
+#include "olap/olap_common.h"
#include "runtime/define_primitive_type.h"
#include "vec/common/cow.h"
#include "vec/common/pod_array_fwd.h"
@@ -129,6 +130,11 @@ public:
return nullptr;
}
+ // Only used on ColumnDictionary
+ virtual void set_rowset_segment_id(std::pair<RowsetId, uint32_t> rowset_segment_id) {}
+
+ virtual std::pair<RowsetId, uint32_t> get_rowset_segment_id() const { return {}; }
+
/// Returns number of values in column.
virtual size_t size() const = 0;
diff --git a/be/src/vec/columns/column_dictionary.h b/be/src/vec/columns/column_dictionary.h
index b8976e77c5..681ed20852 100644
--- a/be/src/vec/columns/column_dictionary.h
+++ b/be/src/vec/columns/column_dictionary.h
@@ -262,6 +262,14 @@ public:
return _dict.find_codes(values, selected);
}
+ void set_rowset_segment_id(std::pair<RowsetId, uint32_t> rowset_segment_id) override {
+ _rowset_segment_id = rowset_segment_id;
+ }
+
+ std::pair<RowsetId, uint32_t> get_rowset_segment_id() const override {
+ return _rowset_segment_id;
+ }
+
bool is_dict_sorted() const { return _dict_sorted; }
bool is_dict_code_converted() const { return _dict_code_converted; }
@@ -451,6 +459,7 @@ private:
Dictionary _dict;
Container _codes;
FieldType _type;
+ std::pair<RowsetId, uint32_t> _rowset_segment_id;
};
template class ColumnDictionary<int32_t>;
diff --git a/be/test/io/cache/remote_file_cache_test.cpp b/be/test/io/cache/remote_file_cache_test.cpp
index a88cf2fbf8..5de0a5f955 100644
--- a/be/test/io/cache/remote_file_cache_test.cpp
+++ b/be/test/io/cache/remote_file_cache_test.cpp
@@ -142,7 +142,7 @@ protected:
EXPECT_EQ("", writer.min_encoded_key().to_string());
EXPECT_EQ("", writer.max_encoded_key().to_string());
- st = segment_v2::Segment::open(fs, path, "", 0, query_schema, res);
+ st = segment_v2::Segment::open(fs, path, "", 0, {}, query_schema, res);
EXPECT_TRUE(st.ok());
EXPECT_EQ(nrows, (*res)->num_rows());
}
diff --git a/be/test/olap/rowset/segment_v2/segment_test.cpp b/be/test/olap/rowset/segment_v2/segment_test.cpp
index ac2d8febfd..e793d4acd3 100644
--- a/be/test/olap/rowset/segment_v2/segment_test.cpp
+++ b/be/test/olap/rowset/segment_v2/segment_test.cpp
@@ -175,7 +175,7 @@ protected:
EXPECT_EQ("", writer.max_encoded_key().to_string());
}
- st = Segment::open(fs, path, "", 0, query_schema, res);
+ st = Segment::open(fs, path, "", 0, {}, query_schema, res);
EXPECT_TRUE(st.ok());
EXPECT_EQ(nrows, (*res)->num_rows());
}
@@ -774,7 +774,7 @@ TEST_F(SegmentReaderWriterTest, TestStringDict) {
{
std::shared_ptr<Segment> segment;
- st = Segment::open(fs, fname, "", 0, tablet_schema, &segment);
+ st = Segment::open(fs, fname, "", 0, {}, tablet_schema, &segment);
EXPECT_TRUE(st.ok());
EXPECT_EQ(4096, segment->num_rows());
Schema schema(tablet_schema);
diff --git a/be/test/tools/benchmark_tool.cpp b/be/test/tools/benchmark_tool.cpp
index b5c1280796..7b6a296e13 100644
--- a/be/test/tools/benchmark_tool.cpp
+++ b/be/test/tools/benchmark_tool.cpp
@@ -364,7 +364,7 @@ public:
writer.finalize(&file_size, &index_size);
file_writer->close();
- Segment::open(fs, path, "", seg_id, &_tablet_schema, res);
+ Segment::open(fs, path, "", seg_id, {}, &_tablet_schema, res);
}
std::vector<std::vector<std::string>> generate_dataset(int rows_number) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/39: [thirdpart](lib) Add lock free queue of concurrentqueue (#14045)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 89951f3f91609f869df033df9ee5a4430a2d468a
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Tue Nov 8 11:34:23 2022 +0800
[thirdpart](lib) Add lock free queue of concurrentqueue (#14045)
---
dist/licenses/LICENSE-concurrentqueue.txt | 21 +++++++++++++++++++++
thirdparty/build-thirdparty.sh | 8 ++++++++
thirdparty/vars.sh | 7 +++++++
3 files changed, 36 insertions(+)
diff --git a/dist/licenses/LICENSE-concurrentqueue.txt b/dist/licenses/LICENSE-concurrentqueue.txt
new file mode 100644
index 0000000000..f5f584534b
--- /dev/null
+++ b/dist/licenses/LICENSE-concurrentqueue.txt
@@ -0,0 +1,21 @@
+This license file applies to everything in this repository except that which is explicitly annotated as being written by other authors, i.e. the Boost queue (included in the benchmarks for comparison), Intel's TBB library (ditto), dlib::pipe (ditto), the CDSChecker tool (used for verification), the Relacy model checker (ditto), and Jeff Preshing's semaphore implementation (used in the blocking queue) which has a zlib license (embedded in lightweightsempahore.h).
+
+Simplified BSD License:
+
+Copyright (c) 2013-2016, Cameron Desrochers. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROF [...]
+
+I have also chosen to dual-license under the Boost Software License as an alternative to the Simplified BSD license above:
+
+Boost Software License - Version 1.0 - August 17th, 2003
+
+Permission is hereby granted, free of charge, to any person or organization obtaining a copy of the software and accompanying documentation covered by this license (the "Software") to use, reproduce, display, distribute, execute, and transmit the Software, and to prepare derivative works of the Software, and to permit third-parties to whom the Software is furnished to do so, all subject to the following:
+
+The copyright notices in the Software and this entire statement, including the above license grant, this restriction and the following disclaimer, must be included in all copies of the Software, in whole or in part, and all derivative works of the Software, unless such copies or derivative works are solely in the form of machine-executable object code generated by a source language processor.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 9d385fe24f..e58d2b0883 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1505,6 +1505,13 @@ build_gettext() {
remove_all_dylib
}
+# concurrentqueue
+build_concurrentqueue() {
+ check_if_source_exist "${CONCURRENTQUEUE_SOURCE}"
+ cd "${TP_SOURCE_DIR}/${CONCURRENTQUEUE_SOURCE}"
+ cp ./*.h "${TP_INSTALL_DIR}/include/"
+}
+
if [[ "$(uname -s)" == 'Darwin' ]]; then
echo 'build for Darwin'
build_binutils
@@ -1565,5 +1572,6 @@ build_opentelemetry
build_libbacktrace
build_sse2neon
build_xxhash
+build_concurrentqueue
echo "Finished to build all thirdparties"
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index b35cc9c127..7097285c75 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -429,6 +429,12 @@ XXHASH_NAME=xxHash-0.8.1.tar.gz
XXHASH_SOURCE=xxHash-0.8.1
XXHASH_MD5SUM="b67c587f5ff4894253da0095ba7ea393"
+# concurrentqueue
+CONCURRENTQUEUE_DOWNLOAD="https://github.com/cameron314/concurrentqueue/archive/refs/tags/v1.0.3.tar.gz"
+CONCURRENTQUEUE_NAME=concurrentqueue-1.0.3.tar.gz
+CONCURRENTQUEUE_SOURCE=concurrentqueue-1.0.3
+CONCURRENTQUEUE_MD5SUM="118e5bb661b567634647312991e10222"
+
# all thirdparties which need to be downloaded is set in array TP_ARCHIVES
export TP_ARCHIVES=(
'LIBEVENT'
@@ -491,6 +497,7 @@ export TP_ARCHIVES=(
'LIBBACKTRACE'
'SSE2NEON'
'XXHASH'
+ 'CONCURRENTQUEUE'
)
if [[ "$(uname -s)" == 'Darwin' ]]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 20/39: [opt](ssb) Add query hint for the SSB queries (#14089)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-unstable
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bd2daec2ef39e8867a76210e1dab8ceb22b3e851
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Nov 9 08:37:31 2022 +0800
[opt](ssb) Add query hint for the SSB queries (#14089)
---
dist/LICENSE-dist.txt | 1 +
tools/ssb-tools/ssb-queries/q4.1.sql | 4 ++--
tools/ssb-tools/ssb-queries/q4.2.sql | 4 ++--
tools/ssb-tools/ssb-queries/q4.3.sql | 4 ++--
4 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/dist/LICENSE-dist.txt b/dist/LICENSE-dist.txt
index feb4183336..69f561132f 100644
--- a/dist/LICENSE-dist.txt
+++ b/dist/LICENSE-dist.txt
@@ -1574,3 +1574,4 @@ Other dependencies:
* breakpad@38ee0be -- license/LICENSE-breakpod.txt
* xsimd: xmid@e9234cd6 -- license/LICENSE-xsimd.txt
* xxhash: 0.8.1 -- license/LICENSE-xxhash.txt
+ * concurrentqueue: 1.0.3 -- license/LICENSE-concurrentqueue.txt
diff --git a/tools/ssb-tools/ssb-queries/q4.1.sql b/tools/ssb-tools/ssb-queries/q4.1.sql
index f0cfcdd403..0cf4f05a64 100644
--- a/tools/ssb-tools/ssb-queries/q4.1.sql
+++ b/tools/ssb-tools/ssb-queries/q4.1.sql
@@ -14,7 +14,7 @@
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.
-SELECT
+SELECT /*+SET_VAR(parallel_fragment_exec_instance_num=4, enable_vectorized_engine=true, batch_size=4096, enable_cost_based_join_reorder=true, enable_projection=true) */
d_year,
c_nation,
SUM(lo_revenue - lo_supplycost) AS PROFIT
@@ -31,4 +31,4 @@ WHERE
OR p_mfgr = 'MFGR#2'
)
GROUP BY d_year, c_nation
-ORDER BY d_year, c_nation;
\ No newline at end of file
+ORDER BY d_year, c_nation;
diff --git a/tools/ssb-tools/ssb-queries/q4.2.sql b/tools/ssb-tools/ssb-queries/q4.2.sql
index fbbaef00e8..97a832c60b 100644
--- a/tools/ssb-tools/ssb-queries/q4.2.sql
+++ b/tools/ssb-tools/ssb-queries/q4.2.sql
@@ -14,7 +14,7 @@
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.
-SELECT
+SELECT /*+SET_VAR(parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, enable_cost_based_join_reorder=true, enable_projection=true) */
d_year,
s_nation,
p_category,
@@ -36,4 +36,4 @@ WHERE
OR p_mfgr = 'MFGR#2'
)
GROUP BY d_year, s_nation, p_category
-ORDER BY d_year, s_nation, p_category;
\ No newline at end of file
+ORDER BY d_year, s_nation, p_category;
diff --git a/tools/ssb-tools/ssb-queries/q4.3.sql b/tools/ssb-tools/ssb-queries/q4.3.sql
index 64582cc6ac..e5b7dc3699 100644
--- a/tools/ssb-tools/ssb-queries/q4.3.sql
+++ b/tools/ssb-tools/ssb-queries/q4.3.sql
@@ -14,7 +14,7 @@
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.
-SELECT
+SELECT /*+SET_VAR(parallel_fragment_exec_instance_num=2, enable_vectorized_engine=true, batch_size=4096, enable_cost_based_join_reorder=true, enable_projection=true) */
d_year,
s_city,
p_brand,
@@ -32,4 +32,4 @@ WHERE
)
AND p_category = 'MFGR#14'
GROUP BY d_year, s_city, p_brand
-ORDER BY d_year, s_city, p_brand;
\ No newline at end of file
+ORDER BY d_year, s_city, p_brand;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org