You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2023/04/23 16:20:33 UTC

[doris] branch branch-2.0-alpha updated (7471ff24f9 -> ca2123cbf6)

This is an automated email from the ASF dual-hosted git repository.

morningman pushed a change to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git


    from 7471ff24f9 [refactor](exceptionsafe) disallow call new method explicitly (#18830)
     new 76d4240a2a [chore](regression) Add 'sync' after stream_load in some cases (#18945)
     new 5f4029466a [Fix](docker) Fix regression test docker issues. (#18928)
     new b3a5ef5d84 [fix](planner) fix pr "using crchash replace murmurhash in the runtime filter" (#18759)
     new 7fb86133e1 [bugfix](asan) fix possible asan check bug in exception to string (#18936)
     new cda8b21ff1 [test](fix) fix postgresql test (#18900)
     new 2eea9f54a3 [Fix](mutli-catalogs) Fix jdbc regression tests. (#18927)
     new eb329be486 [fix](Nereids) two phase read for topn only support simple case (#18955)
     new ca2123cbf6 [Fix](inverted index) fix memory leak when create bkd reader (#18914)

The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 be/src/common/exception.h                          |   5 +-
 be/src/exprs/bloom_filter_func.h                   |  12 +-
 be/src/exprs/runtime_filter.cpp                    |  13 ++-
 be/src/olap/bloom_filter_predicate.h               |  31 +++--
 .../segment_v2/inverted_index_compound_reader.cpp  |  11 ++
 .../segment_v2/inverted_index_compound_reader.h    |   2 +
 .../rowset/segment_v2/inverted_index_reader.cpp    |  12 +-
 be/src/vec/columns/column_dictionary.h             |  29 ++++-
 be/src/vec/exprs/vbloom_predicate.cpp              |   1 +
 .../init/03-create-extension.sql}                  |   2 +-
 .../init/{03-insert.sql => 04-insert.sql}          |   0
 .../postgresql/postgresql-14.yaml.tpl              |   2 +-
 docker/thirdparties/run-thirdparties-docker.sh     |   6 +-
 .../glue/translator/PhysicalPlanTranslator.java    |   6 +-
 .../nereids/processor/post/PlanPostProcessor.java  |   5 +
 .../nereids/processor/post/PlanPostProcessors.java |   2 +-
 .../doris/nereids/processor/post/TopNScanOpt.java  |   2 +-
 .../nereids/processor/post/TwoPhaseReadOpt.java    |  55 ++++++---
 .../nereids/trees/plans/physical/PhysicalTopN.java |   2 +-
 .../nereids/postprocess/TopNRuntimeFilterTest.java |   4 +-
 .../data/nereids_p0/show/test_show_where.out       | 128 +++++++++++++++++++++
 .../data/query_p0/show/test_show_where.out         |   8 ++
 .../correctness_p0/test_current_timestamp.groovy   |   9 ++
 .../jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy |   1 -
 .../show/test_show_create_catalog.groovy           |  47 --------
 .../query_p0/show/test_show_create_catalog.groovy  |  45 --------
 .../test_aggregate_all_functions.groovy            |   2 +
 27 files changed, 293 insertions(+), 149 deletions(-)
 copy docker/thirdparties/docker-compose/{mysql/init/02-create-db.sql => postgresql/init/03-create-extension.sql} (95%)
 rename docker/thirdparties/docker-compose/postgresql/init/{03-insert.sql => 04-insert.sql} (100%)
 delete mode 100644 regression-test/suites/nereids_p0/show/test_show_create_catalog.groovy
 delete mode 100644 regression-test/suites/query_p0/show/test_show_create_catalog.groovy


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 02/08: [Fix](docker) Fix regression test docker issues. (#18928)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 5f4029466a172a51d4f0abb6732280e74bdf349c
Author: Qi Chen <ka...@gmail.com>
AuthorDate: Sat Apr 22 18:03:50 2023 +0800

    [Fix](docker) Fix regression test docker issues. (#18928)
    
    1. Fix not reset data after pg restarted.
    2. 'docker-compose' to 'docker compose'.
---
 docker/thirdparties/run-thirdparties-docker.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docker/thirdparties/run-thirdparties-docker.sh b/docker/thirdparties/run-thirdparties-docker.sh
index ac457cdbfe..ea6f903449 100755
--- a/docker/thirdparties/run-thirdparties-docker.sh
+++ b/docker/thirdparties/run-thirdparties-docker.sh
@@ -190,7 +190,7 @@ if [[ "${RUN_PG}" -eq 1 ]]; then
     sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env down
     if [[ "${STOP}" -ne 1 ]]; then
         sudo mkdir -p "${ROOT}"/docker-compose/postgresql/data/data
-        sudo rm "${ROOT}"/docker-compose/postgresql/data/data/* -rf
+        sudo rm "${ROOT}"/docker-compose/postgresql/data/* -rf
         sudo docker compose -f "${ROOT}"/docker-compose/postgresql/postgresql-14.yaml --env-file "${ROOT}"/docker-compose/postgresql/postgresql-14.env up -d
     fi
 fi
@@ -241,10 +241,10 @@ if [[ "${RUN_HIVE}" -eq 1 ]]; then
     sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hive-2x.yaml
     sed -i "s/doris--/${CONTAINER_UID}/g" "${ROOT}"/docker-compose/hive/hadoop-hive.env.tpl
     sudo bash "${ROOT}"/docker-compose/hive/gen_env.sh
-    sudo docker-compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
+    sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env down
     sudo sed -i '/${CONTAINER_UID}namenode/d' /etc/hosts
     if [[ "${STOP}" -ne 1 ]]; then
-        sudo docker-compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env up --build --remove-orphans -d
+        sudo docker compose -f "${ROOT}"/docker-compose/hive/hive-2x.yaml --env-file "${ROOT}"/docker-compose/hive/hadoop-hive.env up --build --remove-orphans -d
         sudo echo "127.0.0.1 ${CONTAINER_UID}namenode" >> /etc/hosts
     fi
 fi


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 03/08: [fix](planner) fix pr "using crchash replace murmurhash in the runtime filter" (#18759)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit b3a5ef5d840d7f71f332fab773365afda1b197d6
Author: Mryange <59...@users.noreply.github.com>
AuthorDate: Sun Apr 23 10:33:35 2023 +0800

    [fix](planner) fix pr "using crchash replace murmurhash in the runtime filter" (#18759)
---
 be/src/exprs/bloom_filter_func.h       | 12 +++++++++---
 be/src/exprs/runtime_filter.cpp        | 13 +++++++++++--
 be/src/olap/bloom_filter_predicate.h   | 31 +++++++++++++++++++++++--------
 be/src/vec/columns/column_dictionary.h | 29 ++++++++++++++++++++++++++++-
 be/src/vec/exprs/vbloom_predicate.cpp  |  1 +
 5 files changed, 72 insertions(+), 14 deletions(-)

diff --git a/be/src/exprs/bloom_filter_func.h b/be/src/exprs/bloom_filter_func.h
index d8569fbdb5..56bf298a8a 100644
--- a/be/src/exprs/bloom_filter_func.h
+++ b/be/src/exprs/bloom_filter_func.h
@@ -479,9 +479,13 @@ public:
 
     // This function is only to be used if the be_exec_version may be less than 2. If updated, please delete it.
     void insert_crc32_hash(const void* data) override {
-        if constexpr (std::is_same_v<typename BloomFilterTypeTraits<type>::FindOp, StringFindOp>) {
+        if constexpr (std::is_same_v<typename BloomFilterTypeTraits<type>::FindOp, StringFindOp> ||
+                      std::is_same_v<typename BloomFilterTypeTraits<type>::FindOp,
+                                     FixedStringFindOp>) {
             DCHECK(_bloom_filter != nullptr);
             dummy.insert_crc32_hash(*_bloom_filter, data);
+        } else {
+            insert(data);
         }
     }
 
@@ -513,11 +517,13 @@ public:
 
     // This function is only to be used if the be_exec_version may be less than 2. If updated, please delete it.
     bool find_crc32_hash(const void* data) const override {
-        if constexpr (std::is_same_v<typename BloomFilterTypeTraits<type>::FindOp, StringFindOp>) {
+        if constexpr (std::is_same_v<typename BloomFilterTypeTraits<type>::FindOp, StringFindOp> ||
+                      std::is_same_v<typename BloomFilterTypeTraits<type>::FindOp,
+                                     FixedStringFindOp>) {
             DCHECK(_bloom_filter != nullptr);
             return dummy.find_crc32_hash(*_bloom_filter, data);
         }
-        return false;
+        return find(data);
     }
 
     bool find_olap_engine(const void* data) const override {
diff --git a/be/src/exprs/runtime_filter.cpp b/be/src/exprs/runtime_filter.cpp
index 30d8f3b988..43cbd7db33 100644
--- a/be/src/exprs/runtime_filter.cpp
+++ b/be/src/exprs/runtime_filter.cpp
@@ -426,7 +426,12 @@ public:
                 }
             } else {
                 while (it->has_next()) {
-                    bloom_filter->insert(it->get_value());
+                    if (_use_new_hash) {
+                        bloom_filter->insert_crc32_hash(it->get_value());
+                    } else {
+                        bloom_filter->insert(it->get_value());
+                    }
+
                     it->next();
                 }
             }
@@ -449,7 +454,11 @@ public:
             break;
         }
         case RuntimeFilterType::BLOOM_FILTER: {
-            _context.bloom_filter_func->insert(data);
+            if (_use_new_hash) {
+                _context.bloom_filter_func->insert_crc32_hash(data);
+            } else {
+                _context.bloom_filter_func->insert(data);
+            }
             break;
         }
         case RuntimeFilterType::IN_OR_BLOOM_FILTER: {
diff --git a/be/src/olap/bloom_filter_predicate.h b/be/src/olap/bloom_filter_predicate.h
index d845871393..9090ea86a3 100644
--- a/be/src/olap/bloom_filter_predicate.h
+++ b/be/src/olap/bloom_filter_predicate.h
@@ -65,16 +65,31 @@ private:
         uint16_t new_size = 0;
         if (column.is_column_dictionary()) {
             auto* dict_col = reinterpret_cast<const vectorized::ColumnDictI32*>(&column);
-            for (uint16_t i = 0; i < size; i++) {
-                uint16_t idx = sel[i];
-                sel[new_size] = idx;
-                if constexpr (is_nullable) {
-                    new_size += !null_map[idx] &&
-                                _specific_filter->find_uint32_t(dict_col->get_hash_value(idx));
-                } else {
-                    new_size += _specific_filter->find_uint32_t(dict_col->get_hash_value(idx));
+            if (_be_exec_version >= 2) {
+                for (uint16_t i = 0; i < size; i++) {
+                    uint16_t idx = sel[i];
+                    sel[new_size] = idx;
+                    if constexpr (is_nullable) {
+                        new_size += !null_map[idx] && _specific_filter->find_uint32_t(
+                                                              dict_col->get_crc32_hash_value(idx));
+                    } else {
+                        new_size += _specific_filter->find_uint32_t(
+                                dict_col->get_crc32_hash_value(idx));
+                    }
+                }
+            } else {
+                for (uint16_t i = 0; i < size; i++) {
+                    uint16_t idx = sel[i];
+                    sel[new_size] = idx;
+                    if constexpr (is_nullable) {
+                        new_size += !null_map[idx] &&
+                                    _specific_filter->find_uint32_t(dict_col->get_hash_value(idx));
+                    } else {
+                        new_size += _specific_filter->find_uint32_t(dict_col->get_hash_value(idx));
+                    }
                 }
             }
+
         } else if (IRuntimeFilter::enable_use_batch(_be_exec_version, T)) {
             const auto& data =
                     reinterpret_cast<
diff --git a/be/src/vec/columns/column_dictionary.h b/be/src/vec/columns/column_dictionary.h
index c8bf6ea17b..8a2c055d13 100644
--- a/be/src/vec/columns/column_dictionary.h
+++ b/be/src/vec/columns/column_dictionary.h
@@ -288,7 +288,9 @@ public:
     }
 
     uint32_t get_hash_value(uint32_t idx) const { return _dict.get_hash_value(_codes[idx], _type); }
-
+    uint32_t get_crc32_hash_value(uint32_t idx) const {
+        return _dict.get_crc32_hash_value(_codes[idx], _type);
+    }
     template <typename HybridSetType>
     void find_codes(const HybridSetType* values, std::vector<vectorized::UInt8>& selected) const {
         return _dict.find_codes(values, selected);
@@ -396,6 +398,31 @@ public:
             }
         }
 
+        inline uint32_t get_crc32_hash_value(T code, FieldType type) const {
+            if (_compute_hash_value_flags[code]) {
+                return _hash_values[code];
+            } else {
+                auto& sv = (*_dict_data)[code];
+                // The char data is stored in the disk with the schema length,
+                // and zeros are filled if the length is insufficient
+
+                // When reading data, use shrink_char_type_column_suffix_zero(_char_type_idx)
+                // Remove the suffix 0
+                // When writing data, use the CharField::consume function to fill in the trailing 0.
+
+                // For dictionary data of char type, sv.size is the schema length,
+                // so use strnlen to remove the 0 at the end to get the actual length.
+                int32_t len = sv.size;
+                if (type == FieldType::OLAP_FIELD_TYPE_CHAR) {
+                    len = strnlen(sv.data, sv.size);
+                }
+                uint32_t hash_val = HashUtil::crc_hash(sv.data, len, 0);
+                _hash_values[code] = hash_val;
+                _compute_hash_value_flags[code] = 1;
+                return _hash_values[code];
+            }
+        }
+
         // For > , code takes upper_bound - 1; For >= , code takes upper_bound
         // For < , code takes upper_bound; For <=, code takes upper_bound - 1
         // For example a sorted dict: <'b',0> <'c',1> <'d',2>
diff --git a/be/src/vec/exprs/vbloom_predicate.cpp b/be/src/vec/exprs/vbloom_predicate.cpp
index 78f406c8a9..3c2d556bd3 100644
--- a/be/src/vec/exprs/vbloom_predicate.cpp
+++ b/be/src/vec/exprs/vbloom_predicate.cpp
@@ -71,6 +71,7 @@ Status VBloomPredicate::execute(VExprContext* context, Block* block, int* result
         // This is only to be used if the be_exec_version may be less than 2. If updated, please delete it.
         if (_be_exec_version >= 2) {
             for (size_t i = 0; i < sz; i++) {
+                /// TODO: remove virtual function call in get_data_at to improve performance
                 auto ele = argument_column->get_data_at(i);
                 const StringRef v(ele.data, ele.size);
                 ptr[i] = _filter->find_crc32_hash(reinterpret_cast<const void*>(&v));


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 05/08: [test](fix) fix postgresql test (#18900)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit cda8b21ff1e499eff7455072fac6811bc916f69e
Author: yongkang.zhong <zh...@qq.com>
AuthorDate: Sun Apr 23 18:41:41 2023 +0800

    [test](fix) fix postgresql test (#18900)
    
    * [test](fix) fix postgresql test
    
    * fix
---
 .../postgresql/init/03-create-extension.sql            | 18 ++++++++++++++++++
 .../postgresql/init/{03-insert.sql => 04-insert.sql}   |  0
 .../docker-compose/postgresql/postgresql-14.yaml.tpl   |  2 +-
 3 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/docker/thirdparties/docker-compose/postgresql/init/03-create-extension.sql b/docker/thirdparties/docker-compose/postgresql/init/03-create-extension.sql
new file mode 100644
index 0000000000..5237c544f8
--- /dev/null
+++ b/docker/thirdparties/docker-compose/postgresql/init/03-create-extension.sql
@@ -0,0 +1,18 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+CREATE EXTENSION IF NOT EXISTS postgis;
diff --git a/docker/thirdparties/docker-compose/postgresql/init/03-insert.sql b/docker/thirdparties/docker-compose/postgresql/init/04-insert.sql
similarity index 100%
rename from docker/thirdparties/docker-compose/postgresql/init/03-insert.sql
rename to docker/thirdparties/docker-compose/postgresql/init/04-insert.sql
diff --git a/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl b/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
index bfa16e2d52..ae63c2b9bd 100644
--- a/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
+++ b/docker/thirdparties/docker-compose/postgresql/postgresql-14.yaml.tpl
@@ -19,7 +19,7 @@ version: "2.1"
 
 services:
   doris--postgres:
-    image: postgres:14.1
+    image: postgis/postgis:14-3.3
     restart: always
     environment:
       POSTGRES_PASSWORD: 123456


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 06/08: [Fix](mutli-catalogs) Fix jdbc regression tests. (#18927)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 2eea9f54a3b7250407e1e1550ad4f223f1985f6a
Author: Qi Chen <ka...@gmail.com>
AuthorDate: Sun Apr 23 19:42:13 2023 +0800

    [Fix](mutli-catalogs) Fix jdbc regression tests. (#18927)
    
    - Fix `test_show_where` result.
    - Remove `enable_decimal_conversion = true` in `test_mysql_jdbc_catalog`.
    - Remove `test_show_create_catalog`.
---
 .../data/nereids_p0/show/test_show_where.out       | 128 +++++++++++++++++++++
 .../data/query_p0/show/test_show_where.out         |   8 ++
 .../jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy |   1 -
 .../show/test_show_create_catalog.groovy           |  47 --------
 .../query_p0/show/test_show_create_catalog.groovy  |  45 --------
 5 files changed, 136 insertions(+), 93 deletions(-)

diff --git a/regression-test/data/nereids_p0/show/test_show_where.out b/regression-test/data/nereids_p0/show/test_show_where.out
index 6e51af956f..9777dc55f7 100644
--- a/regression-test/data/nereids_p0/show/test_show_where.out
+++ b/regression-test/data/nereids_p0/show/test_show_where.out
@@ -17,3 +17,131 @@ ex_tb1
 ex_tb0
 ex_tb1
 
+-- !select --
+doris_test
+
+-- !select --
+ex_tb0
+ex_tb1
+ex_tb10
+ex_tb11
+ex_tb12
+ex_tb13
+ex_tb14
+ex_tb15
+ex_tb16
+ex_tb17
+ex_tb18
+ex_tb19
+ex_tb2
+ex_tb20
+ex_tb3
+ex_tb4
+ex_tb5
+ex_tb6
+ex_tb7
+ex_tb8
+ex_tb9
+test1
+test_insert
+test_insert2
+
+-- !select --
+ex_tb0
+
+-- !select --
+ex_tb0
+ex_tb1
+ex_tb10
+ex_tb11
+ex_tb12
+ex_tb13
+ex_tb14
+ex_tb15
+ex_tb16
+ex_tb17
+ex_tb18
+ex_tb19
+ex_tb2
+ex_tb20
+ex_tb3
+ex_tb4
+ex_tb5
+ex_tb6
+ex_tb7
+ex_tb8
+ex_tb9
+test1
+test_insert
+test_insert2
+
+-- !select --
+ex_tb0
+ex_tb1
+
+-- !select --
+ex_tb0
+ex_tb1
+ex_tb10
+ex_tb11
+ex_tb12
+ex_tb13
+ex_tb14
+ex_tb15
+ex_tb16
+ex_tb17
+ex_tb18
+ex_tb19
+ex_tb2
+ex_tb20
+ex_tb3
+ex_tb4
+ex_tb5
+ex_tb6
+ex_tb7
+ex_tb8
+ex_tb9
+test1
+test_insert
+test_insert2
+
+-- !select --
+doris_test
+
+-- !select --
+ex_tb0
+ex_tb1
+
+-- !select --
+ex_tb1
+
+-- !select --
+ex_tb0
+ex_tb1
+
+-- !select --
+ex_tb0
+ex_tb1
+ex_tb10
+ex_tb11
+ex_tb12
+ex_tb13
+ex_tb14
+ex_tb15
+ex_tb16
+ex_tb17
+ex_tb18
+ex_tb19
+ex_tb2
+ex_tb20
+ex_tb3
+ex_tb4
+ex_tb5
+ex_tb6
+ex_tb7
+ex_tb8
+ex_tb9
+test1
+test_insert
+test_insert2
+
diff --git a/regression-test/data/query_p0/show/test_show_where.out b/regression-test/data/query_p0/show/test_show_where.out
index 01b4be2c96..9777dc55f7 100644
--- a/regression-test/data/query_p0/show/test_show_where.out
+++ b/regression-test/data/query_p0/show/test_show_where.out
@@ -43,6 +43,8 @@ ex_tb7
 ex_tb8
 ex_tb9
 test1
+test_insert
+test_insert2
 
 -- !select --
 ex_tb0
@@ -70,6 +72,8 @@ ex_tb7
 ex_tb8
 ex_tb9
 test1
+test_insert
+test_insert2
 
 -- !select --
 ex_tb0
@@ -98,6 +102,8 @@ ex_tb7
 ex_tb8
 ex_tb9
 test1
+test_insert
+test_insert2
 
 -- !select --
 doris_test
@@ -136,4 +142,6 @@ ex_tb7
 ex_tb8
 ex_tb9
 test1
+test_insert
+test_insert2
 
diff --git a/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy b/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
index 92785baf80..b0bf171dee 100644
--- a/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
+++ b/regression-test/suites/jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy
@@ -50,7 +50,6 @@ suite("test_mysql_jdbc_catalog", "p0") {
         String test_insert = "test_insert";
         String test_insert2 = "test_insert2";
 
-        sql """ADMIN SET FRONTEND CONFIG ("enable_decimal_conversion" = "true");"""
         sql """drop catalog if exists ${catalog_name} """
         sql """ drop resource if exists ${resource_name} """
 
diff --git a/regression-test/suites/nereids_p0/show/test_show_create_catalog.groovy b/regression-test/suites/nereids_p0/show/test_show_create_catalog.groovy
deleted file mode 100644
index 1bb4540eaf..0000000000
--- a/regression-test/suites/nereids_p0/show/test_show_create_catalog.groovy
+++ /dev/null
@@ -1,47 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements.  See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership.  The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License.  You may obtain a copy of the License at
-//
-//   http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied.  See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-suite("test_show_create_catalog", "query") {
-    sql "SET enable_nereids_planner=true"
-    sql "SET enable_fallback_to_original_planner=false"
-    String catalog_name = "test_show_create_mysql_jdbc_catalog";
-    try {
-        String enabled = context.config.otherConfigs.get("enableJdbcTest")
-        String mysql_port = context.config.otherConfigs.get("mysql_57_port");
-        if (enabled != null && enabled.equalsIgnoreCase("true")) {
-            
-            sql """drop catalog if exists ${catalog_name} """
-
-            // if use 'com.mysql.cj.jdbc.Driver' here, it will report: ClassNotFound
-            sql """ CREATE CATALOG ${catalog_name} PROPERTIES (
-                    "type"="jdbc",
-                    "jdbc.user"="root",
-                    "jdbc.password"="123456",
-                    "jdbc.jdbc_url" = "jdbc:mysql://127.0.0.1:${mysql_port}/doris_test?useSSL=false",
-                    "jdbc.driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",
-                    "jdbc.driver_class" = "com.mysql.cj.jdbc.Driver");
-                """
-
-            qt_select "show create catalog `${catalog_name}`"
-
-        }
-    } finally {
-
-        try_sql("DROP CATALOG IF EXISTS `${catalog_name}`")
-    }
-   
-}
diff --git a/regression-test/suites/query_p0/show/test_show_create_catalog.groovy b/regression-test/suites/query_p0/show/test_show_create_catalog.groovy
deleted file mode 100644
index 14dd568b40..0000000000
--- a/regression-test/suites/query_p0/show/test_show_create_catalog.groovy
+++ /dev/null
@@ -1,45 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements.  See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership.  The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License.  You may obtain a copy of the License at
-//
-//   http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied.  See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-suite("test_show_create_catalog", "query") {
-    String catalog_name = "test_show_create_mysql_jdbc_catalog";
-    try {
-        String enabled = context.config.otherConfigs.get("enableJdbcTest")
-        String mysql_port = context.config.otherConfigs.get("mysql_57_port");
-        if (enabled != null && enabled.equalsIgnoreCase("true")) {
-            
-            sql """drop catalog if exists ${catalog_name} """
-
-            // if use 'com.mysql.cj.jdbc.Driver' here, it will report: ClassNotFound
-            sql """ CREATE CATALOG ${catalog_name} PROPERTIES (
-                    "type"="jdbc",
-                    "jdbc.user"="root",
-                    "jdbc.password"="123456",
-                    "jdbc.jdbc_url" = "jdbc:mysql://127.0.0.1:${mysql_port}/doris_test?useSSL=false",
-                    "jdbc.driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",
-                    "jdbc.driver_class" = "com.mysql.cj.jdbc.Driver");
-                """
-
-            qt_select "show create catalog `${catalog_name}`"
-
-        }
-    } finally {
-
-        try_sql("DROP CATALOG IF EXISTS `${catalog_name}`")
-    }
-   
-}


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 01/08: [chore](regression) Add 'sync' after stream_load in some cases (#18945)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 76d4240a2ab2e41caf9230099ed0782e8f49ecfb
Author: Jerry Hu <mr...@gmail.com>
AuthorDate: Sun Apr 23 14:39:33 2023 +0800

    [chore](regression) Add 'sync' after stream_load in some cases (#18945)
---
 .../suites/correctness_p0/test_current_timestamp.groovy          | 9 +++++++++
 .../aggregate_functions/test_aggregate_all_functions.groovy      | 2 ++
 2 files changed, 11 insertions(+)

diff --git a/regression-test/suites/correctness_p0/test_current_timestamp.groovy b/regression-test/suites/correctness_p0/test_current_timestamp.groovy
index 8aca63fc06..209ce465e4 100644
--- a/regression-test/suites/correctness_p0/test_current_timestamp.groovy
+++ b/regression-test/suites/correctness_p0/test_current_timestamp.groovy
@@ -78,6 +78,9 @@ suite("test_current_timestamp") {
 
         time 10000 // limit inflight 10s
     }
+
+    sql "sync"
+
     qt_stream_load_csv1 """ select count(*) from ${tableName} where id > 4 and to_date(dt_0) = to_date(dt_1); """
     qt_stream_load_csv2 """ select count(*) from ${tableName} where id > 4 and to_date(dt_2) = to_date(dt_3); """
     qt_stream_load_csv3 """ select count(*) from ${tableName} where id > 4 and to_date(dt_4) = to_date(dt_5); """
@@ -98,6 +101,9 @@ suite("test_current_timestamp") {
 
         time 10000 // limit inflight 10s
     }
+
+    sql "sync"
+
     qt_stream_load_json1 """ select count(*) from ${tableName} where id > 8 and to_date(dt_0) = to_date(dt_1); """
     qt_stream_load_json2 """ select count(*) from ${tableName} where id > 8 and to_date(dt_2) = to_date(dt_3); """
     qt_stream_load_json3 """ select count(*) from ${tableName} where id > 8 and to_date(dt_4) = to_date(dt_5); """
@@ -117,6 +123,9 @@ suite("test_current_timestamp") {
         file 'test_current_timestamp_streamload.json'
         time 10000 // limit inflight 10s
     }
+
+    sql "sync"
+
     qt_stream_load_json5 """ select id, name, dt_1  from ${tableName2} order by id; """
     qt_stream_load_json6 """ select count(*) from ${tableName2} where dt_2 is not NULL; """
  }
diff --git a/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_all_functions.groovy b/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_all_functions.groovy
index 973ea22dff..dbc6a998a5 100644
--- a/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_all_functions.groovy
+++ b/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_all_functions.groovy
@@ -530,6 +530,8 @@ suite("test_aggregate_all_functions") {
         inputIterator rows.iterator()
     }
 
+    sql "sync"
+
     qt_select48 """select dt, id, quantile_percent(quantile_union(price), 0) from ${tableName_21} group by dt, id order by dt, id"""
 
     qt_select49 """select dt, id, quantile_percent(quantile_union(price), 0.5) from ${tableName_21} group by dt, id order by dt, id"""


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 08/08: [Fix](inverted index) fix memory leak when create bkd reader (#18914)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit ca2123cbf66ff3aa3bdfd8b282b03bcf20b38dd6
Author: airborne12 <ai...@gmail.com>
AuthorDate: Sun Apr 23 23:21:44 2023 +0800

    [Fix](inverted index) fix memory leak when create bkd reader (#18914)
    
    The function compoundReader->openInput is called three times, and if any of these calls fail,
    an error is logged, and the function returns early. If one or two of the calls succeed, but the others fail,
    there might be a situation where the allocated memory for the IndexInput objects is not freed.
    
    To fix this, you could use std::unique_ptr to manage the memory for IndexInput objects.
    This would automatically clean up the memory when the function goes out of scope.
---
 .../rowset/segment_v2/inverted_index_compound_reader.cpp     | 11 +++++++++++
 .../olap/rowset/segment_v2/inverted_index_compound_reader.h  |  2 ++
 be/src/olap/rowset/segment_v2/inverted_index_reader.cpp      | 12 ++++++------
 3 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.cpp b/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.cpp
index 9ce20703b4..7748d105cc 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.cpp
@@ -224,6 +224,17 @@ int64_t DorisCompoundReader::fileLength(const char* name) const {
     return e->length;
 }
 
+bool DorisCompoundReader::openInput(const char* name,
+                                    std::unique_ptr<lucene::store::IndexInput>& ret,
+                                    CLuceneError& error, int32_t bufferSize) {
+    lucene::store::IndexInput* tmp;
+    bool success = openInput(name, tmp, error, bufferSize);
+    if (success) {
+        ret.reset(tmp);
+    }
+    return success;
+}
+
 bool DorisCompoundReader::openInput(const char* name, lucene::store::IndexInput*& ret,
                                     CLuceneError& error, int32_t bufferSize) {
     if (stream == nullptr) {
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.h b/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.h
index d6e95d9d50..a400737b34 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.h
+++ b/be/src/olap/rowset/segment_v2/inverted_index_compound_reader.h
@@ -67,6 +67,8 @@ public:
     int64_t fileLength(const char* name) const override;
     bool openInput(const char* name, lucene::store::IndexInput*& ret, CLuceneError& err,
                    int32_t bufferSize = -1) override;
+    bool openInput(const char* name, std::unique_ptr<lucene::store::IndexInput>& ret,
+                   CLuceneError& err, int32_t bufferSize = -1);
     void renameFile(const char* from, const char* to) override;
     void touchFile(const char* name) override;
     lucene::store::IndexOutput* createOutput(const char* name) override;
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp b/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
index 9a7520dc57..68508274c3 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
@@ -557,9 +557,9 @@ Status BkdIndexReader::get_bkd_reader(std::shared_ptr<lucene::util::bkd::bkd_rea
         return Status::Error<ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND>();
     }
     CLuceneError err;
-    lucene::store::IndexInput* data_in;
-    lucene::store::IndexInput* meta_in;
-    lucene::store::IndexInput* index_in;
+    std::unique_ptr<lucene::store::IndexInput> data_in;
+    std::unique_ptr<lucene::store::IndexInput> meta_in;
+    std::unique_ptr<lucene::store::IndexInput> index_in;
 
     if (!compoundReader->openInput(
                 InvertedIndexDescriptor::get_temporary_bkd_index_data_file_name().c_str(), data_in,
@@ -574,12 +574,12 @@ Status BkdIndexReader::get_bkd_reader(std::shared_ptr<lucene::util::bkd::bkd_rea
         return Status::Error<ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND>();
     }
 
-    bkdReader = std::make_shared<lucene::util::bkd::bkd_reader>(data_in);
-    if (0 == bkdReader->read_meta(meta_in)) {
+    bkdReader = std::make_shared<lucene::util::bkd::bkd_reader>(data_in.release());
+    if (0 == bkdReader->read_meta(meta_in.get())) {
         return Status::EndOfFile("bkd index file is empty");
     }
 
-    bkdReader->read_index(index_in);
+    bkdReader->read_index(index_in.get());
 
     _type_info = get_scalar_type_info((FieldType)bkdReader->type);
     if (_type_info == nullptr) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 07/08: [fix](Nereids) two phase read for topn only support simple case (#18955)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit eb329be48606600241a6903be9349f818496901f
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Sun Apr 23 21:32:23 2023 +0800

    [fix](Nereids) two phase read for topn only support simple case (#18955)
    
    1. topn must has merge node
    2. topn must the top node of plan
---
 .../glue/translator/PhysicalPlanTranslator.java    |  6 +--
 .../nereids/processor/post/PlanPostProcessor.java  |  5 ++
 .../nereids/processor/post/PlanPostProcessors.java |  2 +-
 .../doris/nereids/processor/post/TopNScanOpt.java  |  2 +-
 .../nereids/processor/post/TwoPhaseReadOpt.java    | 55 +++++++++++++++-------
 .../nereids/trees/plans/physical/PhysicalTopN.java |  2 +-
 .../nereids/postprocess/TopNRuntimeFilterTest.java |  4 +-
 7 files changed, 49 insertions(+), 27 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index 4b80f64036..aa0dc972c1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -884,7 +884,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
             SortNode sortNode = translateSortNode(topN, inputFragment.getPlanRoot(), context);
             sortNode.setOffset(topN.getOffset());
             sortNode.setLimit(topN.getLimit());
-            if (topN.getMutableState(PhysicalTopN.TOPN_OPT).isPresent()) {
+            if (topN.getMutableState(PhysicalTopN.TOPN_RUNTIME_FILTER).isPresent()) {
                 sortNode.setUseTopnOpt(true);
                 PlanNode child = sortNode.getChild(0);
                 Preconditions.checkArgument(child instanceof OlapScanNode,
@@ -928,9 +928,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
         });
         List<Expr> sortTupleOutputList = new ArrayList<>();
         List<Slot> outputList = sort.getOutput();
-        outputList.forEach(k -> {
-            sortTupleOutputList.add(ExpressionTranslator.translate(k, context));
-        });
+        outputList.forEach(k -> sortTupleOutputList.add(ExpressionTranslator.translate(k, context)));
         // 2. Generate new Tuple and get current slotRef for newOrderingExprList
         List<Expr> newOrderingExprList = Lists.newArrayList();
         TupleDescriptor tupleDesc = generateTupleDesc(outputList, orderKeyList, newOrderingExprList, context, null);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessor.java
index fa6a9deaa9..5090acedf4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessor.java
@@ -18,10 +18,15 @@
 package org.apache.doris.nereids.processor.post;
 
 import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.trees.plans.Plan;
 import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
 
 /**
  * PlanPostprocessor: a PlanVisitor to rewrite PhysicalPlan to new PhysicalPlan.
  */
 public class PlanPostProcessor extends DefaultPlanRewriter<CascadesContext> {
+
+    public Plan processRoot(Plan plan, CascadesContext ctx) {
+        return plan.accept(this, ctx);
+    }
 }
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
index b96e7bbf3e..0843d1e04b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
@@ -47,7 +47,7 @@ public class PlanPostProcessors {
     public PhysicalPlan process(PhysicalPlan physicalPlan) {
         PhysicalPlan resultPlan = physicalPlan;
         for (PlanPostProcessor processor : getProcessors()) {
-            resultPlan = (PhysicalPlan) resultPlan.accept(processor, cascadesContext);
+            resultPlan = (PhysicalPlan) processor.processRoot(resultPlan, cascadesContext);
         }
         return resultPlan;
     }
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TopNScanOpt.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TopNScanOpt.java
index bb00983d2d..a938a231ed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TopNScanOpt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TopNScanOpt.java
@@ -78,7 +78,7 @@ public class TopNScanOpt extends PlanPostProcessor {
         olapScan = (PhysicalOlapScan) child;
 
         if (olapScan.getTable().isDupKeysOrMergeOnWrite()) {
-            topN.setMutableState(PhysicalTopN.TOPN_OPT, true);
+            topN.setMutableState(PhysicalTopN.TOPN_RUNTIME_FILTER, true);
         }
 
         return topN;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TwoPhaseReadOpt.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TwoPhaseReadOpt.java
index 454caae435..543f908456 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TwoPhaseReadOpt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/TwoPhaseReadOpt.java
@@ -28,6 +28,7 @@ import org.apache.doris.nereids.trees.plans.Plan;
 import org.apache.doris.nereids.trees.plans.SortPhase;
 import org.apache.doris.nereids.trees.plans.algebra.Filter;
 import org.apache.doris.nereids.trees.plans.algebra.Project;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute;
 import org.apache.doris.nereids.trees.plans.physical.PhysicalFilter;
 import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapScan;
 import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
@@ -52,23 +53,40 @@ import java.util.Set;
 public class TwoPhaseReadOpt extends PlanPostProcessor {
 
     @Override
-    public PhysicalTopN visitPhysicalTopN(PhysicalTopN<? extends Plan> topN, CascadesContext ctx) {
-        topN.child().accept(this, ctx);
-        Plan child = topN.child();
-        if (topN.getSortPhase() != SortPhase.LOCAL_SORT) {
-            return topN;
+    public Plan processRoot(Plan plan, CascadesContext ctx) {
+        if (plan instanceof PhysicalTopN) {
+            PhysicalTopN<Plan> physicalTopN = (PhysicalTopN<Plan>) plan;
+            if (physicalTopN.getSortPhase() == SortPhase.MERGE_SORT) {
+                return plan.accept(this, ctx);
+            }
         }
-        if (topN.getOrderKeys().isEmpty()) {
-            return topN;
+        return plan;
+    }
+
+    @Override
+    public PhysicalTopN visitPhysicalTopN(PhysicalTopN<? extends Plan> mergeTopN, CascadesContext ctx) {
+        mergeTopN.child().accept(this, ctx);
+        Plan child = mergeTopN.child();
+        if (mergeTopN.getSortPhase() != SortPhase.MERGE_SORT || !(mergeTopN.child() instanceof PhysicalDistribute)) {
+            return mergeTopN;
+        }
+        PhysicalDistribute<Plan> distribute = (PhysicalDistribute<Plan>) mergeTopN.child();
+        if (!(distribute.child() instanceof PhysicalTopN)) {
+            return mergeTopN;
+        }
+        PhysicalTopN<Plan> localTopN = (PhysicalTopN<Plan>) distribute.child();
+
+        if (localTopN.getOrderKeys().isEmpty()) {
+            return mergeTopN;
         }
 
         // topn opt
         long topNOptLimitThreshold = getTopNOptLimitThreshold();
-        if (topNOptLimitThreshold < 0 || topN.getLimit() > topNOptLimitThreshold) {
-            return topN;
+        if (topNOptLimitThreshold < 0 || mergeTopN.getLimit() > topNOptLimitThreshold) {
+            return mergeTopN;
         }
-        if (!topN.getOrderKeys().stream().map(OrderKey::getExpr).allMatch(Expression::isColumnFromTable)) {
-            return topN;
+        if (!localTopN.getOrderKeys().stream().map(OrderKey::getExpr).allMatch(Expression::isColumnFromTable)) {
+            return mergeTopN;
         }
 
         PhysicalOlapScan olapScan;
@@ -81,18 +99,18 @@ public class TwoPhaseReadOpt extends PlanPostProcessor {
             if (child instanceof Project) {
                 project = (PhysicalProject<Plan>) child;
                 // TODO: remove this after fix two phase read on project core
-                return topN;
+                return mergeTopN;
             }
             child = child.child(0);
         }
         if (!(child instanceof PhysicalOlapScan)) {
-            return topN;
+            return mergeTopN;
         }
         olapScan = (PhysicalOlapScan) child;
 
         // all order key must column from table
         if (!olapScan.getTable().getEnableLightSchemaChange()) {
-            return topN;
+            return mergeTopN;
         }
 
         Map<ExprId, ExprId> projectRevertedMap = Maps.newHashMap();
@@ -114,22 +132,23 @@ public class TwoPhaseReadOpt extends PlanPostProcessor {
         if (filter != null) {
             filter.getConjuncts().forEach(e -> deferredMaterializedExprIds.removeAll(e.getInputSlotExprIds()));
         }
-        topN.getOrderKeys().stream()
+        localTopN.getOrderKeys().stream()
                 .map(OrderKey::getExpr)
                 .map(Slot.class::cast)
                 .map(NamedExpression::getExprId)
                 .map(projectRevertedMap::get)
                 .filter(Objects::nonNull)
                 .forEach(deferredMaterializedExprIds::remove);
-        topN.getOrderKeys().stream()
+        localTopN.getOrderKeys().stream()
                 .map(OrderKey::getExpr)
                 .map(Slot.class::cast)
                 .map(NamedExpression::getExprId)
                 .forEach(deferredMaterializedExprIds::remove);
-        topN.setMutableState(PhysicalTopN.TWO_PHASE_READ_OPT, true);
+        localTopN.setMutableState(PhysicalTopN.TWO_PHASE_READ_OPT, true);
+        mergeTopN.setMutableState(PhysicalTopN.TWO_PHASE_READ_OPT, true);
         olapScan.setMutableState(PhysicalOlapScan.DEFERRED_MATERIALIZED_SLOTS, deferredMaterializedExprIds);
 
-        return topN;
+        return mergeTopN;
     }
 
     private long getTopNOptLimitThreshold() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
index 2ca9a4c51c..4a58f5d9e9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalTopN.java
@@ -40,7 +40,7 @@ import java.util.Optional;
  */
 public class PhysicalTopN<CHILD_TYPE extends Plan> extends AbstractPhysicalSort<CHILD_TYPE> implements TopN {
 
-    public static final String TOPN_OPT = "topn_opt";
+    public static final String TOPN_RUNTIME_FILTER = "topn_runtime_filter";
     public static final String TWO_PHASE_READ_OPT = "two_phase_read_opt";
 
     private final long limit;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/TopNRuntimeFilterTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/TopNRuntimeFilterTest.java
index 9da544143a..6cbbcbf071 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/TopNRuntimeFilterTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/TopNRuntimeFilterTest.java
@@ -42,7 +42,7 @@ public class TopNRuntimeFilterTest extends SSBTestBase {
         new PlanPostProcessors(checker.getCascadesContext()).process(plan);
         Assertions.assertTrue(plan.children().get(0) instanceof PhysicalTopN);
         PhysicalTopN localTopN = (PhysicalTopN) plan.children().get(0);
-        Assertions.assertTrue(localTopN.getMutableState(PhysicalTopN.TOPN_OPT).isPresent());
+        Assertions.assertTrue(localTopN.getMutableState(PhysicalTopN.TOPN_RUNTIME_FILTER).isPresent());
     }
 
     // topn rf do not apply on string-like and float column
@@ -56,6 +56,6 @@ public class TopNRuntimeFilterTest extends SSBTestBase {
         new PlanPostProcessors(checker.getCascadesContext()).process(plan);
         Assertions.assertTrue(plan.children().get(0) instanceof PhysicalTopN);
         PhysicalTopN localTopN = (PhysicalTopN) plan.children().get(0);
-        Assertions.assertFalse(localTopN.getMutableState(PhysicalTopN.TOPN_OPT).isPresent());
+        Assertions.assertFalse(localTopN.getMutableState(PhysicalTopN.TOPN_RUNTIME_FILTER).isPresent());
     }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


[doris] 04/08: [bugfix](asan) fix possible asan check bug in exception to string (#18936)

Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch branch-2.0-alpha
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 7fb86133e13aab7f3c795f900921c6f71002c894
Author: yiguolei <67...@qq.com>
AuthorDate: Sun Apr 23 12:26:36 2023 +0800

    [bugfix](asan) fix possible asan check bug in exception to string (#18936)
    
    Co-authored-by: yiguolei <yi...@gmail.com>
---
 be/src/common/exception.h | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/be/src/common/exception.h b/be/src/common/exception.h
index 3723db0891..29437cfead 100644
--- a/be/src/common/exception.h
+++ b/be/src/common/exception.h
@@ -45,10 +45,7 @@ public:
     Exception(int code, const std::string_view fmt, Args&&... args)
             : Exception(code, fmt::format(fmt, std::forward<Args>(args)...)) {}
 
-    std::string code_as_string() const {
-        return (int)_code >= 0 ? doris::to_string(static_cast<TStatusCode::type>(_code))
-                               : fmt::format("E{}", (int16_t)_code);
-    }
+    std::string code_as_string() const { return fmt::format("E{}", _code); }
 
     int code() const { return _code; }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org