You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/01/30 07:33:58 UTC

(doris) branch branch-2.1 updated (6eba030897d -> 7fe4d00bb21)

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git


    from 6eba030897d [fix](chore) path gc should consider tablet migration (#30095) (#30548)
     new 49d17f2be2b [fix](move-memtable) fix potential duplicate of TabletStream profile (#30397)
     new f0a35f6e2d5 [regression test](schema change) add some case for agg col (#30479)
     new f7e01ceffa8 [bug](node) add dependency for set operation node (#30203)
     new 57a8c75ddcc [regression test](schema change) add case for column type change (#30472)
     new 123a187c239 [regression test](schema change) add case for modify partition info (#30338)
     new 5f20d7c5d00 [regression test](stream load) test for `enable_profile` (#28534)
     new 009bca96524 [regression test](broker load) add partition load case (#28259)
     new 589e83561c6 [Fix](jdk17) support start FE with JDK17 (#29658) (#30485)
     new 129463f557c [Try_Fix](scan) try fix the scanner schedule logic to prevent excessive memory usage and timeout (#30515)
     new 28c4e691491 [fix](move-memtable) check load timeout before close wait (#30526)
     new 0932dadcff7 [enhancement](log) print detail error for segment compaction failure (#30503)
     new 08897b7e031 [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor #30529
     new b712f0b8102 [improvement](index)add index_id column in show index stmt (#30431)
     new 5731ed7aad1 [fix](Nereids): add order by when testing pkfk to avoid unstable res #30507
     new 4648902350e [bugfix](iceberg)fix read NULL with date partition (#30478)
     new 53c624ffa09 [feat](Nereids): support alias when eliminate join for partially mv rewritting #30498
     new 9e63015e2fd [fix](mtmv)resolve task tvf concurrent modification exception #30497
     new c810df55c50 [fix](Nereids) create constraint write edit log in lock scope (#30422)
     new 218fb80938e [fix](group commit) Fix group commit VOlapTablePartitionParam memory … (#30491)
     new b7c86d66faa [improvement](thirdparty) introduce brpc stream patch to make stream write to socket in background bthread (#30458)
     new 12827ceb16c [fix](Nereids): fix wrong regression test (#30520)
     new ea427e8c51f [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30541)
     new 11f7b36dab8 [bugfix](paimon)add class loader (#30483)
     new 7fe4d00bb21 [fix](regex) use boost regex instead of std (#30462)

The 24 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 be/src/olap/delete_handler.cpp                     |  12 +-
 be/src/olap/memtable_flush_executor.cpp            |  63 +++++---
 be/src/olap/memtable_flush_executor.h              |  21 ++-
 be/src/olap/memtable_writer.cpp                    |   6 +-
 be/src/olap/rowset/beta_rowset_writer.cpp          |  11 +-
 be/src/olap/rowset/segcompaction.cpp               |   3 +-
 be/src/pipeline/pipeline_fragment_context.cpp      |   6 +-
 be/src/runtime/load_stream.cpp                     |   2 +-
 be/src/vec/exec/scan/pip_scanner_context.h         |  14 +-
 be/src/vec/exec/scan/scanner_context.cpp           |   6 +-
 be/src/vec/exec/scan/scanner_context.h             |   5 +-
 be/src/vec/exec/vset_operation_node.cpp            |  11 +-
 be/src/vec/sink/group_commit_block_sink.cpp        |   2 +-
 be/src/vec/sink/group_commit_block_sink.h          |   2 +-
 be/src/vec/sink/writer/vtablet_writer_v2.cpp       |   5 +
 be/test/runtime/load_stream_test.cpp               |   3 +-
 bin/start_be.sh                                    |   7 +-
 bin/start_fe.sh                                    |   8 +-
 conf/be.conf                                       |   3 +
 conf/fe.conf                                       |   4 +-
 .../org/apache/doris/paimon/PaimonJniScanner.java  |  20 ++-
 .../apache/doris/alter/SchemaChangeHandler.java    |   2 +-
 .../org/apache/doris/analysis/ShowIndexStmt.java   |   1 +
 .../java/org/apache/doris/catalog/TableIf.java     |  76 ++++-----
 .../apache/doris/job/extensions/mtmv/MTMVJob.java  |   2 +-
 .../java/org/apache/doris/mtmv/MTMVJobInfo.java    |  11 +-
 .../jobs/joinorder/hypergraph/GraphSimplifier.java |   3 +-
 .../jobs/joinorder/hypergraph/HyperGraph.java      |  39 ++++-
 .../jobs/joinorder/hypergraph/edge/Edge.java       |   4 +
 .../jobs/joinorder/hypergraph/edge/JoinEdge.java   |  19 ++-
 .../rules/exploration/mv/HyperGraphComparator.java | 116 ++++++++------
 .../trees/plans/commands/AddConstraintCommand.java |  13 +-
 .../plans/commands/DropConstraintCommand.java      |   5 +-
 .../org/apache/doris/persist/gson/GsonUtils.java   |   2 +
 .../planner/external/iceberg/IcebergScanNode.java  |  19 ++-
 .../java/org/apache/doris/qe/ShowExecutor.java     |   3 +-
 .../catalog/constraint/ConstraintPersistTest.java  |   7 +-
 .../rules/exploration/mv/EliminateJoinTest.java    |  25 ++-
 fe/pom.xml                                         |   2 +-
 .../test_external_catalog_iceberg_partition.out    |  33 ++++
 .../data/index_p0/test_bitmap_index.out            |  58 -------
 .../test_decimal_bitmap_index_multi_page.out       |   3 -
 .../data/inverted_index_p0/test_bitmap_index.out   |  40 -----
 .../data/inverted_index_p0/test_inverted_index.out |  52 -------
 ...th_merge.csv => broker_load_with_partition.csv} |   6 +-
 .../test_broker_load_with_partition.out            |  10 ++
 .../data/load_p0/http_stream/test_http_stream.out  |   4 +
 .../data/nereids_rules_p0/pkfk/eliminate_inner.out |   8 +-
 .../push_down_top_n_distinct_through_join.out      |   2 +-
 .../data/nereids_syntax_p0/set_operation.out       |   4 +
 .../test_schema_change_agg_check_all_types.out     |  27 ++++
 regression-test/pipeline/p0/conf/be.conf           |   3 +
 regression-test/pipeline/p0/conf/fe.conf           |   3 +
 .../test_external_catalog_iceberg_partition.groovy |  21 +++
 .../suites/index_p0/test_bitmap_index.groovy       |  68 ++++++++-
 .../test_decimal_bitmap_index_multi_page.groovy    |   5 +-
 .../inverted_index_p0/test_bitmap_index.groovy     |  46 +++++-
 .../inverted_index_p0/test_inverted_index.groovy   |  58 ++++++-
 ...oovy => test_broker_load_with_partition.groovy} |  46 +++---
 .../load_p0/http_stream/test_http_stream.groovy    |  39 +++++
 .../nereids_rules_p0/pkfk/eliminate_inner.groovy   |   4 +-
 .../push_down_top_n_distinct_through_join.groovy   |   4 +-
 .../suites/nereids_syntax_p0/set_operation.groovy  |  74 +++++++++
 .../suites/schema_change_p0/test_partition.groovy  |  29 ++++
 .../schema_change_p0/test_schema_change_agg.groovy | 161 +++++++++----------
 .../test_schema_change_agg_check_all_types.groovy  | 170 ++++++++-------------
 .../patches/brpc-1.4.0-write-background.patch      | 170 +++++++++++++++++++++
 67 files changed, 1137 insertions(+), 574 deletions(-)
 copy regression-test/data/load_p0/broker_load/{broker_load_with_merge.csv => broker_load_with_partition.csv} (50%)
 create mode 100644 regression-test/data/load_p0/broker_load/test_broker_load_with_partition.out
 copy regression-test/suites/load_p0/broker_load/{test_broker_load_seq.groovy => test_broker_load_with_partition.groovy} (76%)
 create mode 100644 thirdparty/patches/brpc-1.4.0-write-background.patch


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 02/24: [regression test](schema change) add some case for agg col (#30479)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit f0a35f6e2d5c04634028a8644adecce8d6b31a91
Author: Guangdong Liu <li...@gmail.com>
AuthorDate: Mon Jan 29 21:04:04 2024 +0800

    [regression test](schema change) add some case for agg col (#30479)
---
 .../apache/doris/alter/SchemaChangeHandler.java    |   2 +-
 .../schema_change_p0/test_schema_change_agg.groovy | 161 ++++++++++-----------
 2 files changed, 77 insertions(+), 86 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
index 206f028b023..81e8984780e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
@@ -2644,7 +2644,7 @@ public class SchemaChangeHandler extends AlterHandler {
         try {
             changedIndexIdToSchema = checkTable(db, olapTable, indexSchemaMap);
         } catch (DdlException e) {
-            throw new DdlException("Table " + db.getFullName() + "." + olapTable.getName() + " check failed");
+            throw new DdlException("Table " + db.getFullName() + "." + olapTable.getName() + " check failed", e);
         }
 
         if (changedIndexIdToSchema.isEmpty() && !hasIndexChange) {
diff --git a/regression-test/suites/schema_change_p0/test_schema_change_agg.groovy b/regression-test/suites/schema_change_p0/test_schema_change_agg.groovy
index 2534c219ca2..e2931bda9ac 100644
--- a/regression-test/suites/schema_change_p0/test_schema_change_agg.groovy
+++ b/regression-test/suites/schema_change_p0/test_schema_change_agg.groovy
@@ -15,32 +15,9 @@
 // specific language governing permissions and limitations
 // under the License.
 
-import java.util.Date
-import java.text.SimpleDateFormat
-import org.apache.http.HttpResponse
-import org.apache.http.client.methods.HttpPut
-import org.apache.http.impl.client.CloseableHttpClient
-import org.apache.http.impl.client.HttpClients
-import org.apache.http.entity.ContentType
-import org.apache.http.entity.StringEntity
-import org.apache.http.client.config.RequestConfig
-import org.apache.http.client.RedirectStrategy
-import org.apache.http.protocol.HttpContext
-import org.apache.http.HttpRequest
-import org.apache.http.impl.client.LaxRedirectStrategy
-import org.apache.http.client.methods.RequestBuilder
-import org.apache.http.entity.StringEntity
-import org.apache.http.client.methods.CloseableHttpResponse
-import org.apache.http.util.EntityUtils
-
 suite("test_schema_change_agg", "p0") {
     def tableName3 = "test_all_agg"
 
-    def getJobState = { tableName ->
-        def jobStateResult = sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
-        return jobStateResult[0][9]
-    }
-
     def getCreateViewState = { tableName ->
         def createViewStateResult = sql """ SHOW ALTER TABLE MATERIALIZED VIEW WHERE IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
         return createViewStateResult[0][8]
@@ -96,21 +73,10 @@ suite("test_schema_change_agg", "p0") {
     execStreamLoad()
 
     sql """ alter table ${tableName3} modify column k2 bigint(11) key NULL"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            execStreamLoad()
-            if (max_try_num < 1) {
-                println "test timeout," + "state:" + res
-                assertEquals("FINISHED",res)
-            }
-        }
+
+    waitForSchemaChangeDone {
+        sql """SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1"""
+        time 60
     }
 
     /*
@@ -134,21 +100,9 @@ suite("test_schema_change_agg", "p0") {
     */
 
     sql """ alter table ${tableName3} modify column k4 bigint(11) sum NULL"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            execStreamLoad()
-            if (max_try_num < 1) {
-                println "test timeout," + "state:" + res
-                assertEquals("FINISHED",res)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1"""
+        time 60
     }
 
     sql """ alter table ${tableName3} add column v14 int sum NOT NULL default "0" after k13 """
@@ -156,24 +110,9 @@ suite("test_schema_change_agg", "p0") {
     'a', 'b', 'c', '2021-10-30', '2021-10-30 00:00:00', 10086) """
 
     sql """ alter table ${tableName3} modify column v14 int sum NULL default "0" """
-    sleep(10)
-    max_try_num = 6000
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            int val = 100000 + max_try_num
-            sql """ insert into ${tableName3} values (${val}, 2, 3, 4, 5, 6.6, 1.7, 8.8,
-    'a', 'b', 'c', '2021-10-30', '2021-10-30 00:00:00', 9527) """
-            sleep(10)
-            if (max_try_num < 1) {
-                println "test timeout," + "state:" + res
-                assertEquals("FINISHED",res)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1"""
+        time 60
     }
 
     sql """ alter table ${tableName3} drop column v14 """
@@ -195,20 +134,9 @@ suite("test_schema_change_agg", "p0") {
     // boolean type
     sql """ alter table ${tableName3} add column v15 boolean replace NOT NULL default "0" after k13 """
 
-    sleep(10)
-    max_try_num = 6000
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(2000)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1"""
+        time 60
     }
 
     sql """ insert into ${tableName3} values (10002, 2, 3, 4, 5, 6.6, 1.7, 8.81,
@@ -219,5 +147,68 @@ suite("test_schema_change_agg", "p0") {
         exception "Can not change BOOLEAN to INT"
     }
 
+    // check add column without agg type
+    test {
+        sql """ alter table ${tableName3} add column v16 int NOT NULL default "0" after k13 """
+        exception "Invalid column order. value should be after key. index[${tableName3}]"
+    }
+
+    // del key
+    test {
+        sql """ alter table ${tableName3} drop column k1 """
+        exception "Can not drop key column when table has value column with REPLACE aggregation method"
+    }
+
+
+    //drop partition key
+    sql """ DROP TABLE IF EXISTS ${tableName3} """
+    sql """
+        CREATE TABLE `${tableName3}`
+        (
+            `siteid` INT DEFAULT '10',
+            `citycode` SMALLINT,
+            `username` VARCHAR(32) DEFAULT 'test',
+            `pv` BIGINT SUM DEFAULT '0'
+        )
+        AGGREGATE KEY(`siteid`, `citycode`, `username`)
+        PARTITION BY RANGE(`siteid`)
+                (
+                    partition `old_p1` values [("1"), ("2")),
+                    partition `old_p2` values [("2"), ("3"))
+                )
+        DISTRIBUTED BY HASH(pv) BUCKETS 1
+        PROPERTIES (
+            "replication_num" = "1"
+        );
+    """
+
+    test {
+        sql "alter table ${tableName3} drop column siteid"
+        exception "Partition column[siteid] cannot be dropped. index[${tableName3}]"
+    }
+
+    //modify col
+
+    // without agg type
+    test {
+        sql "alter table ${tableName3} modify column pv varchar"
+        exception "Can not change aggregation type"
+    }
+
+    //partition col
+    test {
+        sql "alter table ${tableName3} modify column siteid varchar DEFAULT '10'"
+        exception "Can not modify partition column[siteid]."
+    }
+
+    //distribution key
+
+    test {
+        sql "alter table ${tableName3} modify column pv bigint sum default '0' comment 'pv'"
+        exception "Can not modify distribution column[pv]. index[${tableName3}]"
+    }
+
+
+
 }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 16/24: [feat](Nereids): support alias when eliminate join for partially mv rewritting #30498

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 53c624ffa0924545ae870e190910a6e83840ca7b
Author: 谢健 <ji...@gmail.com>
AuthorDate: Tue Jan 30 11:53:07 2024 +0800

    [feat](Nereids): support alias when eliminate join for partially mv rewritting #30498
---
 .../jobs/joinorder/hypergraph/GraphSimplifier.java |   3 +-
 .../jobs/joinorder/hypergraph/HyperGraph.java      |  39 ++++++-
 .../jobs/joinorder/hypergraph/edge/Edge.java       |   4 +
 .../jobs/joinorder/hypergraph/edge/JoinEdge.java   |  19 +++-
 .../rules/exploration/mv/HyperGraphComparator.java | 116 +++++++++++++--------
 .../rules/exploration/mv/EliminateJoinTest.java    |  25 ++++-
 6 files changed, 154 insertions(+), 52 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/GraphSimplifier.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/GraphSimplifier.java
index 5539e894cc7..61738f2fca6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/GraphSimplifier.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/GraphSimplifier.java
@@ -33,6 +33,7 @@ import org.apache.doris.nereids.util.JoinUtils;
 import org.apache.doris.statistics.Statistics;
 
 import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableSet;
 
 import java.util.ArrayDeque;
 import java.util.ArrayList;
@@ -417,7 +418,7 @@ public class GraphSimplifier {
 
         JoinEdge newEdge = new JoinEdge(join, edge.getIndex(),
                 edge.getLeftChildEdges(), edge.getRightChildEdges(), edge.getSubTreeNodes(),
-                edge.getLeftRequiredNodes(), edge.getRightRequiredNodes());
+                edge.getLeftRequiredNodes(), edge.getRightRequiredNodes(), ImmutableSet.of(), ImmutableSet.of());
         newEdge.addLeftExtendNode(leftNodes);
         newEdge.addRightExtendNode(rightNodes);
         return newEdge;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
index 8472837d79e..0099ab30afc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/HyperGraph.java
@@ -46,14 +46,17 @@ import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
 
 import java.util.ArrayList;
 import java.util.BitSet;
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.Optional;
 import java.util.Set;
 import java.util.stream.Collectors;
+import javax.annotation.Nullable;
 
 /**
  * The graph is a join graph, whose node is the leaf plan and edge is a join operator.
@@ -288,6 +291,36 @@ public class HyperGraph {
         return new HyperGraph.Builder().buildHyperGraphForMv(plan);
     }
 
+    /**
+     * map output to requires output and construct named expressions
+     */
+    public @Nullable List<NamedExpression> getNamedExpressions(
+            long nodeMap, Set<Slot> outputSet, Set<Slot> requireOutputs) {
+        List<NamedExpression> output = new ArrayList<>();
+        List<NamedExpression> projects = getComplexProject().get(nodeMap);
+        if (projects == null) {
+            return null;
+        }
+        for (Slot slot : requireOutputs) {
+            if (outputSet.contains(slot)) {
+                output.add(slot);
+            } else {
+                Optional<NamedExpression> expr = projects.stream()
+                        .filter(p -> p.toSlot().equals(slot))
+                        .findFirst();
+                if (!expr.isPresent()) {
+                    return null;
+                }
+                // TODO: consider cascades alias
+                if (!outputSet.containsAll(expr.get().getInputSlots())) {
+                    return null;
+                }
+                output.add(expr.get());
+            }
+        }
+        return output;
+    }
+
     /**
      * Builder of HyperGraph
      */
@@ -509,6 +542,10 @@ public class HyperGraph {
             }
 
             BitSet curJoinEdges = new BitSet();
+            Set<Slot> leftInputSlots = ImmutableSet.copyOf(
+                    Sets.intersection(join.getInputSlots(), join.left().getOutputSet()));
+            Set<Slot> rightInputSlots = ImmutableSet.copyOf(
+                    Sets.intersection(join.getInputSlots(), join.right().getOutputSet()));
             for (Map.Entry<Pair<Long, Long>, Pair<List<Expression>, List<Expression>>> entry : conjuncts
                     .entrySet()) {
                 LogicalJoin<?, ?> singleJoin = new LogicalJoin<>(join.getJoinType(), entry.getValue().first,
@@ -518,7 +555,7 @@ public class HyperGraph {
                 Pair<Long, Long> ends = entry.getKey();
                 JoinEdge edge = new JoinEdge(singleJoin, joinEdges.size(), leftEdgeNodes.first, rightEdgeNodes.first,
                         LongBitmap.newBitmapUnion(leftEdgeNodes.second, rightEdgeNodes.second),
-                        ends.first, ends.second);
+                        ends.first, ends.second, leftInputSlots, rightInputSlots);
                 for (int nodeIndex : LongBitmap.getIterator(edge.getReferenceNodes())) {
                     nodes.get(nodeIndex).attachEdge(edge);
                 }
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/Edge.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/Edge.java
index 2530074931e..c41d9e270ac 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/Edge.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/Edge.java
@@ -79,6 +79,10 @@ public abstract class Edge {
         return LongBitmap.getCardinality(leftExtendedNodes) == 1 && LongBitmap.getCardinality(rightExtendedNodes) == 1;
     }
 
+    public boolean isRightSimple() {
+        return LongBitmap.getCardinality(rightExtendedNodes) == 1;
+    }
+
     public void addLeftRejectEdge(JoinEdge edge) {
         leftRejectEdges.add(edge);
     }
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/JoinEdge.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/JoinEdge.java
index c23be5f16eb..94f4b30e8d4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/JoinEdge.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/joinorder/hypergraph/edge/JoinEdge.java
@@ -37,12 +37,16 @@ import javax.annotation.Nullable;
 public class JoinEdge extends Edge {
 
     private final LogicalJoin<? extends Plan, ? extends Plan> join;
+    private final Set<Slot> leftInputSlots;
+    private final Set<Slot> rightInputSlots;
 
     public JoinEdge(LogicalJoin<? extends Plan, ? extends Plan> join, int index,
             BitSet leftChildEdges, BitSet rightChildEdges, long subTreeNodes,
-            long leftRequireNodes, long rightRequireNodes) {
+            long leftRequireNodes, long rightRequireNodes, Set<Slot> leftInputSlots, Set<Slot> rightInputSlots) {
         super(index, leftChildEdges, rightChildEdges, subTreeNodes, leftRequireNodes, rightRequireNodes);
         this.join = join;
+        this.leftInputSlots = leftInputSlots;
+        this.rightInputSlots = rightInputSlots;
     }
 
     /**
@@ -51,7 +55,8 @@ public class JoinEdge extends Edge {
     public JoinEdge swap() {
         JoinEdge swapEdge = new
                 JoinEdge(join.swap(), getIndex(), getRightChildEdges(),
-                getLeftChildEdges(), getSubTreeNodes(), getRightRequiredNodes(), getLeftRequiredNodes());
+                getLeftChildEdges(), getSubTreeNodes(), getRightRequiredNodes(), getLeftRequiredNodes(),
+                this.rightInputSlots, this.leftInputSlots);
         swapEdge.addLeftRejectEdges(getLeftRejectEdge());
         swapEdge.addRightRejectEdges(getRightRejectEdge());
         return swapEdge;
@@ -63,7 +68,7 @@ public class JoinEdge extends Edge {
 
     public JoinEdge withJoinTypeAndCleanCR(JoinType joinType) {
         return new JoinEdge(join.withJoinType(joinType), getIndex(), getLeftChildEdges(), getRightChildEdges(),
-                getSubTreeNodes(), getLeftRequiredNodes(), getRightRequiredNodes());
+                getSubTreeNodes(), getLeftRequiredNodes(), getRightRequiredNodes(), leftInputSlots, rightInputSlots);
     }
 
     public LogicalJoin<? extends Plan, ? extends Plan> getJoin() {
@@ -112,4 +117,12 @@ public class JoinEdge extends Edge {
         join.getExpressions().forEach(expression -> slots.addAll(expression.getInputSlots()));
         return slots;
     }
+
+    public Set<Slot> getLeftInputSlots() {
+        return leftInputSlots;
+    }
+
+    public Set<Slot> getRightInputSlots() {
+        return rightInputSlots;
+    }
 }
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
index 3339d009c79..341caa88094 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
@@ -27,9 +27,11 @@ import org.apache.doris.nereids.jobs.joinorder.hypergraph.edge.JoinEdge;
 import org.apache.doris.nereids.jobs.joinorder.hypergraph.node.StructInfoNode;
 import org.apache.doris.nereids.rules.rewrite.PushDownFilterThroughJoin;
 import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
 import org.apache.doris.nereids.trees.expressions.Slot;
 import org.apache.doris.nereids.trees.plans.JoinType;
 import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
 import org.apache.doris.nereids.util.ExpressionUtils;
 import org.apache.doris.nereids.util.JoinUtils;
 
@@ -46,6 +48,7 @@ import java.util.Map;
 import java.util.Map.Entry;
 import java.util.Set;
 import java.util.stream.Collectors;
+import javax.annotation.Nullable;
 
 /**
  * HyperGraphComparator
@@ -144,60 +147,81 @@ public class HyperGraphComparator {
         return buildComparisonRes();
     }
 
-    private boolean tryEliminateNodesAndEdge() {
-        boolean hasFilterEdgeAbove = viewHyperGraph.getFilterEdges().stream()
-                .filter(e -> LongBitmap.getCardinality(e.getReferenceNodes()) == 1)
-                .anyMatch(e -> LongBitmap.isSubset(e.getReferenceNodes(), eliminateViewNodesMap));
-        if (hasFilterEdgeAbove) {
-            // If there is some filter edge above the eliminated node, we should rebuild a plan
-            // Right now, just refuse it.
+    private @Nullable Plan constructViewPlan(long nodeBitmap, Set<Slot> requireOutputs) {
+        if (LongBitmap.getCardinality(nodeBitmap) != 1) {
+            return null;
+        }
+        Plan basePlan = viewHyperGraph.getNode(LongBitmap.lowestOneIndex(nodeBitmap)).getPlan();
+        if (basePlan.getOutputSet().containsAll(requireOutputs)) {
+            return basePlan;
+        }
+        List<NamedExpression> projects = viewHyperGraph
+                .getNamedExpressions(nodeBitmap, basePlan.getOutputSet(), requireOutputs);
+        if (projects == null) {
+            return null;
+        }
+        return new LogicalProject<>(projects, basePlan);
+    }
+
+    private boolean canEliminatePrimaryByForeign(long primaryNodes, long foreignNodes,
+            Set<Slot> primarySlots, Set<Slot> foreignSlots, JoinEdge joinEdge) {
+        Plan foreign = constructViewPlan(foreignNodes, foreignSlots);
+        Plan primary = constructViewPlan(primaryNodes, primarySlots);
+        if (foreign == null || primary == null) {
             return false;
         }
-        for (JoinEdge joinEdge : viewHyperGraph.getJoinEdges()) {
-            if (!LongBitmap.isOverlap(joinEdge.getReferenceNodes(), eliminateViewNodesMap)) {
-                continue;
+        return JoinUtils.canEliminateByFk(joinEdge.getJoin(), primary, foreign);
+    }
+
+    private boolean canEliminateViewEdge(JoinEdge joinEdge) {
+        // eliminate by unique
+        if (joinEdge.getJoinType().isLeftOuterJoin() && joinEdge.isRightSimple()) {
+            long eliminatedRight =
+                    LongBitmap.newBitmapIntersect(joinEdge.getRightExtendedNodes(), eliminateViewNodesMap);
+            if (LongBitmap.getCardinality(eliminatedRight) != 1) {
+                return false;
             }
-            // eliminate by unique
-            if (joinEdge.getJoinType().isLeftOuterJoin()) {
-                long eliminatedRight =
-                        LongBitmap.newBitmapIntersect(joinEdge.getRightExtendedNodes(), eliminateViewNodesMap);
-                if (LongBitmap.getCardinality(eliminatedRight) != 1) {
-                    return false;
-                }
-                Plan rigthPlan = viewHyperGraph
-                        .getNode(LongBitmap.lowestOneIndex(joinEdge.getRightExtendedNodes())).getPlan();
-                return JoinUtils.canEliminateByLeft(joinEdge.getJoin(),
-                        rigthPlan.getLogicalProperties().getFunctionalDependencies());
+            Plan rigthPlan = constructViewPlan(joinEdge.getRightExtendedNodes(), joinEdge.getRightInputSlots());
+            if (rigthPlan == null) {
+                return false;
             }
-            // eliminate by pk fk
-            if (joinEdge.getJoinType().isInnerJoin()) {
-                if (!joinEdge.isSimple()) {
-                    return false;
-                }
-                long eliminatedLeft =
-                        LongBitmap.newBitmapIntersect(joinEdge.getLeftExtendedNodes(), eliminateViewNodesMap);
-                long eliminatedRight =
-                        LongBitmap.newBitmapIntersect(joinEdge.getRightExtendedNodes(), eliminateViewNodesMap);
-                if (LongBitmap.getCardinality(eliminatedLeft) == 0
-                        && LongBitmap.getCardinality(eliminatedRight) == 1) {
-                    Plan foreign = viewHyperGraph
-                            .getNode(LongBitmap.lowestOneIndex(joinEdge.getLeftExtendedNodes())).getPlan();
-                    Plan primary = viewHyperGraph
-                            .getNode(LongBitmap.lowestOneIndex(joinEdge.getRightExtendedNodes())).getPlan();
-                    return JoinUtils.canEliminateByFk(joinEdge.getJoin(), primary, foreign);
-                } else if (LongBitmap.getCardinality(eliminatedLeft) == 1
-                        && LongBitmap.getCardinality(eliminatedRight) == 0) {
-                    Plan foreign = viewHyperGraph
-                            .getNode(LongBitmap.lowestOneIndex(joinEdge.getRightExtendedNodes())).getPlan();
-                    Plan primary = viewHyperGraph
-                            .getNode(LongBitmap.lowestOneIndex(joinEdge.getLeftExtendedNodes())).getPlan();
-                    return JoinUtils.canEliminateByFk(joinEdge.getJoin(), primary, foreign);
-                }
+            return JoinUtils.canEliminateByLeft(joinEdge.getJoin(),
+                    rigthPlan.getLogicalProperties().getFunctionalDependencies());
+        }
+        // eliminate by pk fk
+        if (joinEdge.getJoinType().isInnerJoin()) {
+            if (!joinEdge.isSimple()) {
                 return false;
             }
+            long eliminatedLeft =
+                    LongBitmap.newBitmapIntersect(joinEdge.getLeftExtendedNodes(), eliminateViewNodesMap);
+            long eliminatedRight =
+                    LongBitmap.newBitmapIntersect(joinEdge.getRightExtendedNodes(), eliminateViewNodesMap);
+            if (LongBitmap.getCardinality(eliminatedLeft) == 0
+                    && LongBitmap.getCardinality(eliminatedRight) == 1) {
+                return canEliminatePrimaryByForeign(joinEdge.getRightExtendedNodes(), joinEdge.getLeftExtendedNodes(),
+                        joinEdge.getRightInputSlots(), joinEdge.getLeftInputSlots(), joinEdge);
+            } else if (LongBitmap.getCardinality(eliminatedLeft) == 1
+                    && LongBitmap.getCardinality(eliminatedRight) == 0) {
+                return canEliminatePrimaryByForeign(joinEdge.getLeftExtendedNodes(), joinEdge.getRightExtendedNodes(),
+                        joinEdge.getLeftInputSlots(), joinEdge.getRightInputSlots(), joinEdge);
+            }
+        }
+        return false;
+    }
 
+    private boolean tryEliminateNodesAndEdge() {
+        boolean hasFilterEdgeAbove = viewHyperGraph.getFilterEdges().stream()
+                .filter(e -> LongBitmap.getCardinality(e.getReferenceNodes()) == 1)
+                .anyMatch(e -> LongBitmap.isSubset(e.getReferenceNodes(), eliminateViewNodesMap));
+        if (hasFilterEdgeAbove) {
+            // If there is some filter edge above the eliminated node, we should rebuild a plan
+            // Right now, just reject it.
+            return false;
         }
-        return true;
+        return viewHyperGraph.getJoinEdges().stream()
+                .filter(joinEdge -> LongBitmap.isOverlap(joinEdge.getReferenceNodes(), eliminateViewNodesMap))
+                .allMatch(this::canEliminateViewEdge);
     }
 
     private boolean compareNodeWithExpr(StructInfoNode query, StructInfoNode view) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/EliminateJoinTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/EliminateJoinTest.java
index 3e245741178..cc8b7147423 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/EliminateJoinTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/EliminateJoinTest.java
@@ -52,10 +52,22 @@ class EliminateJoinTest extends SqlTestBase {
                 .rewrite()
                 .applyExploration(RuleSet.BUSHY_TREE_JOIN_REORDER)
                 .getAllPlan().get(0).child(0);
+        CascadesContext c3 = createCascadesContext(
+                "select * from T1 left outer join (select id as id2 from T2 group by id) T2 "
+                        + "on T1.id = T2.id2 ",
+                connectContext
+        );
+        Plan p3 = PlanChecker.from(c3)
+                .analyze()
+                .rewrite()
+                .applyExploration(RuleSet.BUSHY_TREE_JOIN_REORDER)
+                .getAllPlan().get(0).child(0);
         HyperGraph h1 = HyperGraph.builderForMv(p1).buildAll().get(0);
         HyperGraph h2 = HyperGraph.builderForMv(p2).buildAll().get(0);
+        HyperGraph h3 = HyperGraph.builderForMv(p3).buildAll().get(0);
         ComparisonResult res = HyperGraphComparator.isLogicCompatible(h1, h2, constructContext(p1, p2));
         Assertions.assertTrue(!res.isInvalid());
+        Assertions.assertTrue(!HyperGraphComparator.isLogicCompatible(h1, h3, constructContext(p1, p2)).isInvalid());
         Assertions.assertTrue(res.getViewExpressions().isEmpty());
     }
 
@@ -112,11 +124,23 @@ class EliminateJoinTest extends SqlTestBase {
                 .rewrite()
                 .applyExploration(RuleSet.BUSHY_TREE_JOIN_REORDER)
                 .getAllPlan().get(0).child(0);
+        CascadesContext c3 = createCascadesContext(
+                "select * from T1 inner join (select id as id2 from T2) T2 "
+                        + "on T1.id = T2.id2 ",
+                connectContext
+        );
+        Plan p3 = PlanChecker.from(c3)
+                .analyze()
+                .rewrite()
+                .applyExploration(RuleSet.BUSHY_TREE_JOIN_REORDER)
+                .getAllPlan().get(0).child(0);
         HyperGraph h1 = HyperGraph.builderForMv(p1).buildAll().get(0);
         HyperGraph h2 = HyperGraph.builderForMv(p2).buildAll().get(0);
+        HyperGraph h3 = HyperGraph.builderForMv(p3).buildAll().get(0);
         ComparisonResult res = HyperGraphComparator.isLogicCompatible(h1, h2, constructContext(p1, p2));
         Assertions.assertTrue(!res.isInvalid());
         Assertions.assertTrue(res.getViewExpressions().isEmpty());
+        Assertions.assertTrue(!HyperGraphComparator.isLogicCompatible(h1, h3, constructContext(p1, p2)).isInvalid());
         dropConstraint("alter table T2 drop constraint pk");
     }
 
@@ -154,7 +178,6 @@ class EliminateJoinTest extends SqlTestBase {
         dropConstraint("alter table T3 drop constraint uk");
     }
 
-    @Disabled
     @Test
     void testLOJWithPKFKAndUK2() throws Exception {
         connectContext.getSessionVariable().setDisableNereidsRules("INFER_PREDICATES");


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 12/24: [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor #30529

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 08897b7e03135d2e390f3fbba40d531e387e91ae
Author: wangbo <wa...@apache.org>
AuthorDate: Tue Jan 30 10:12:31 2024 +0800

    [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor #30529
---
 be/src/olap/memtable_flush_executor.cpp | 63 +++++++++++++++++++++------------
 be/src/olap/memtable_flush_executor.h   | 21 +++++++----
 be/src/olap/memtable_writer.cpp         |  6 ++--
 3 files changed, 58 insertions(+), 32 deletions(-)

diff --git a/be/src/olap/memtable_flush_executor.cpp b/be/src/olap/memtable_flush_executor.cpp
index 7190597b54c..60ef194aae5 100644
--- a/be/src/olap/memtable_flush_executor.cpp
+++ b/be/src/olap/memtable_flush_executor.cpp
@@ -56,8 +56,7 @@ public:
     ~MemtableFlushTask() override { g_flush_task_num << -1; }
 
     void run() override {
-        _flush_token->_flush_memtable(_memtable.get(), _segment_id, _submit_task_time);
-        _memtable.reset();
+        _flush_token->_flush_memtable(std::move(_memtable), _segment_id, _submit_task_time);
     }
 
 private:
@@ -94,16 +93,35 @@ Status FlushToken::submit(std::unique_ptr<MemTable> mem_table) {
     int64_t submit_task_time = MonotonicNanos();
     auto task = std::make_shared<MemtableFlushTask>(
             this, std::move(mem_table), _rowset_writer->allocate_segment_id(), submit_task_time);
-    _stats.flush_running_count++;
-    return _flush_token->submit(std::move(task));
+    Status ret = _thread_pool->submit(std::move(task));
+    if (ret.ok()) {
+        _stats.flush_running_count++;
+    }
+    return ret;
+}
+
+// NOTE: FlushToken's submit/cancel/wait run in one thread,
+// so we don't need to make them mutually exclusive, std::atomic is enough.
+void FlushToken::_wait_running_task_finish() {
+    while (true) {
+        int64_t flush_running_count = _stats.flush_running_count.load();
+        if (flush_running_count < 0) {
+            LOG(ERROR) << "flush_running_count < 0, this is not expected!";
+        }
+        if (flush_running_count == 0) {
+            break;
+        }
+        std::this_thread::sleep_for(std::chrono::milliseconds(50));
+    }
 }
 
 void FlushToken::cancel() {
-    _flush_token->shutdown();
+    _shutdown_flush_token();
+    _wait_running_task_finish();
 }
 
 Status FlushToken::wait() {
-    _flush_token->wait();
+    _wait_running_task_finish();
     {
         std::shared_lock rdlk(_flush_status_lock);
         if (!_flush_status.ok()) {
@@ -134,8 +152,12 @@ Status FlushToken::_do_flush_memtable(MemTable* memtable, int32_t segment_id, in
     return Status::OK();
 }
 
-void FlushToken::_flush_memtable(MemTable* mem_table, int32_t segment_id,
+void FlushToken::_flush_memtable(std::unique_ptr<MemTable> memtable_ptr, int32_t segment_id,
                                  int64_t submit_task_time) {
+    Defer defer {[&]() { _stats.flush_running_count--; }};
+    if (_is_shutdown()) {
+        return;
+    }
     uint64_t flush_wait_time_ns = MonotonicNanos() - submit_task_time;
     _stats.flush_wait_time_ns += flush_wait_time_ns;
     // If previous flush has failed, return directly
@@ -148,10 +170,10 @@ void FlushToken::_flush_memtable(MemTable* mem_table, int32_t segment_id,
 
     MonotonicStopWatch timer;
     timer.start();
-    size_t memory_usage = mem_table->memory_usage();
+    size_t memory_usage = memtable_ptr->memory_usage();
 
     int64_t flush_size;
-    Status s = _do_flush_memtable(mem_table, segment_id, &flush_size);
+    Status s = _do_flush_memtable(memtable_ptr.get(), segment_id, &flush_size);
 
     {
         std::shared_lock rdlk(_flush_status_lock);
@@ -174,8 +196,7 @@ void FlushToken::_flush_memtable(MemTable* mem_table, int32_t segment_id,
                   << ", mem size: " << memory_usage << ", disk size: " << flush_size;
     _stats.flush_time_ns += timer.elapsed_time();
     _stats.flush_finish_count++;
-    _stats.flush_running_count--;
-    _stats.flush_size_bytes += mem_table->memory_usage();
+    _stats.flush_size_bytes += memtable_ptr->memory_usage();
     _stats.flush_disk_size_bytes += flush_size;
 }
 
@@ -199,27 +220,25 @@ void MemTableFlushExecutor::init(const std::vector<DataDir*>& data_dirs) {
 
 // NOTE: we use SERIAL mode here to ensure all mem-tables from one tablet are flushed in order.
 Status MemTableFlushExecutor::create_flush_token(std::unique_ptr<FlushToken>& flush_token,
-                                                 RowsetWriter* rowset_writer, bool should_serial,
+                                                 RowsetWriter* rowset_writer,
                                                  bool is_high_priority) {
     if (!is_high_priority) {
-        if (rowset_writer->type() == BETA_ROWSET && !should_serial) {
+        if (rowset_writer->type() == BETA_ROWSET) {
             // beta rowset can be flush in CONCURRENT, because each memtable using a new segment writer.
-            flush_token = std::make_unique<FlushToken>(
-                    _flush_pool->new_token(ThreadPool::ExecutionMode::CONCURRENT));
+            flush_token = std::make_unique<FlushToken>(_flush_pool.get());
         } else {
             // alpha rowset do not support flush in CONCURRENT.
-            flush_token = std::make_unique<FlushToken>(
-                    _flush_pool->new_token(ThreadPool::ExecutionMode::SERIAL));
+            // and not support alpha rowset now.
+            return Status::InternalError<false>("not support alpha rowset load now.");
         }
     } else {
-        if (rowset_writer->type() == BETA_ROWSET && !should_serial) {
+        if (rowset_writer->type() == BETA_ROWSET) {
             // beta rowset can be flush in CONCURRENT, because each memtable using a new segment writer.
-            flush_token = std::make_unique<FlushToken>(
-                    _high_prio_flush_pool->new_token(ThreadPool::ExecutionMode::CONCURRENT));
+            flush_token = std::make_unique<FlushToken>(_high_prio_flush_pool.get());
         } else {
             // alpha rowset do not support flush in CONCURRENT.
-            flush_token = std::make_unique<FlushToken>(
-                    _high_prio_flush_pool->new_token(ThreadPool::ExecutionMode::SERIAL));
+            // and not support alpha rowset now.
+            return Status::InternalError<false>("not support alpha rowset load now.");
         }
     }
     flush_token->set_rowset_writer(rowset_writer);
diff --git a/be/src/olap/memtable_flush_executor.h b/be/src/olap/memtable_flush_executor.h
index a025bf43cf8..80983baa66f 100644
--- a/be/src/olap/memtable_flush_executor.h
+++ b/be/src/olap/memtable_flush_executor.h
@@ -38,7 +38,7 @@ class RowsetWriter;
 // use atomic because it may be updated by multi threads
 struct FlushStatistic {
     std::atomic_uint64_t flush_time_ns = 0;
-    std::atomic_uint64_t flush_running_count = 0;
+    std::atomic_int64_t flush_running_count = 0;
     std::atomic_uint64_t flush_finish_count = 0;
     std::atomic_uint64_t flush_size_bytes = 0;
     std::atomic_uint64_t flush_disk_size_bytes = 0;
@@ -56,8 +56,8 @@ std::ostream& operator<<(std::ostream& os, const FlushStatistic& stat);
 //    because the entire job will definitely fail;
 class FlushToken {
 public:
-    explicit FlushToken(std::unique_ptr<ThreadPoolToken> flush_pool_token)
-            : _flush_token(std::move(flush_pool_token)), _flush_status(Status::OK()) {}
+    explicit FlushToken(ThreadPool* thread_pool)
+            : _flush_status(Status::OK()), _thread_pool(thread_pool) {}
 
     Status submit(std::unique_ptr<MemTable> mem_table);
 
@@ -75,15 +75,19 @@ public:
 
     const MemTableStat& memtable_stat() { return _memtable_stat; }
 
+private:
+    void _shutdown_flush_token() { _shutdown.store(true); }
+    bool _is_shutdown() { return _shutdown.load(); }
+    void _wait_running_task_finish();
+
 private:
     friend class MemtableFlushTask;
 
-    void _flush_memtable(MemTable* mem_table, int32_t segment_id, int64_t submit_task_time);
+    void _flush_memtable(std::unique_ptr<MemTable> memtable_ptr, int32_t segment_id,
+                         int64_t submit_task_time);
 
     Status _do_flush_memtable(MemTable* memtable, int32_t segment_id, int64_t* flush_size);
 
-    std::unique_ptr<ThreadPoolToken> _flush_token;
-
     // Records the current flush status of the tablet.
     // Note: Once its value is set to Failed, it cannot return to SUCCESS.
     std::shared_mutex _flush_status_lock;
@@ -94,6 +98,9 @@ private:
     RowsetWriter* _rowset_writer = nullptr;
 
     MemTableStat _memtable_stat;
+
+    std::atomic<bool> _shutdown = false;
+    ThreadPool* _thread_pool = nullptr;
 };
 
 // MemTableFlushExecutor is responsible for flushing memtables to disk.
@@ -119,7 +126,7 @@ public:
     void init(const std::vector<DataDir*>& data_dirs);
 
     Status create_flush_token(std::unique_ptr<FlushToken>& flush_token, RowsetWriter* rowset_writer,
-                              bool should_serial, bool is_high_priority);
+                              bool is_high_priority);
 
 private:
     void _register_metrics();
diff --git a/be/src/olap/memtable_writer.cpp b/be/src/olap/memtable_writer.cpp
index c967cdbd483..9eb44af903f 100644
--- a/be/src/olap/memtable_writer.cpp
+++ b/be/src/olap/memtable_writer.cpp
@@ -76,9 +76,9 @@ Status MemTableWriter::init(std::shared_ptr<RowsetWriter> rowset_writer,
     // create flush handler
     // by assigning segment_id to memtable before submiting to flush executor,
     // we can make sure same keys sort in the same order in all replicas.
-    bool should_serial = false;
-    RETURN_IF_ERROR(StorageEngine::instance()->memtable_flush_executor()->create_flush_token(
-            _flush_token, _rowset_writer.get(), should_serial, _req.is_high_priority));
+    RETURN_IF_ERROR(
+            ExecEnv::GetInstance()->storage_engine().memtable_flush_executor()->create_flush_token(
+                    _flush_token, _rowset_writer.get(), _req.is_high_priority));
 
     _is_init = true;
     return Status::OK();


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 07/24: [regression test](broker load) add partition load case (#28259)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 009bca96524e4f7a1e3d9201c01e33e28ed446a5
Author: Guangdong Liu <li...@gmail.com>
AuthorDate: Mon Jan 29 21:58:31 2024 +0800

    [regression test](broker load) add partition load case (#28259)
---
 .../broker_load/broker_load_with_partition.csv     |   6 +
 .../test_broker_load_with_partition.out            |  10 ++
 .../test_broker_load_with_partition.groovy         | 122 +++++++++++++++++++++
 3 files changed, 138 insertions(+)

diff --git a/regression-test/data/load_p0/broker_load/broker_load_with_partition.csv b/regression-test/data/load_p0/broker_load/broker_load_with_partition.csv
new file mode 100644
index 00000000000..e665e0ae67d
--- /dev/null
+++ b/regression-test/data/load_p0/broker_load/broker_load_with_partition.csv
@@ -0,0 +1,6 @@
+11001,2023-09-01,1,1,10
+11001,2023-09-01,2,1,10
+11001,2023-09-01,1,2,10
+11001,2023-10-01,2,2,10
+11001,2023-10-01,1,3,10
+11001,2023-10-01,2,3,10
diff --git a/regression-test/data/load_p0/broker_load/test_broker_load_with_partition.out b/regression-test/data/load_p0/broker_load/test_broker_load_with_partition.out
new file mode 100644
index 00000000000..5015d98b3c9
--- /dev/null
+++ b/regression-test/data/load_p0/broker_load/test_broker_load_with_partition.out
@@ -0,0 +1,10 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select --
+1	\N	1	1	1
+11001	2023-10-01	1	3	10
+11001	2023-10-01	2	2	10
+11001	2023-10-01	2	3	10
+11001	2023-09-01	1	1	10
+11001	2023-09-01	1	2	10
+11001	2023-09-01	2	1	10
+
diff --git a/regression-test/suites/load_p0/broker_load/test_broker_load_with_partition.groovy b/regression-test/suites/load_p0/broker_load/test_broker_load_with_partition.groovy
new file mode 100644
index 00000000000..032b48baf60
--- /dev/null
+++ b/regression-test/suites/load_p0/broker_load/test_broker_load_with_partition.groovy
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_broker_load_with_partition", "load_p0") {
+    // define a sql table
+    def testTable = "tbl_test_broker_load_with_partition"
+
+    def create_test_table = {testTablex ->
+        def result1 = sql """
+            CREATE TABLE IF NOT EXISTS ${testTable} (
+                `k1` BIGINT NOT NULL,
+                `k2` DATE NULL,
+                `k3` INT(11) NOT NULL,
+                `k4` INT(11) NOT NULL,
+                `v5` BIGINT SUM NULL DEFAULT "0"
+            ) ENGINE=OLAP
+            AGGREGATE KEY(`k1`, `k2`, `k3`, `k4`)
+            COMMENT 'OLAP'
+            PARTITION BY RANGE(`k2`)
+            (
+                PARTITION `p202309` VALUES LESS THAN ("2023-10-01"),
+                PARTITION `p202310` VALUES LESS THAN ("2023-11-01")
+            )
+            DISTRIBUTED BY HASH(`k1`) BUCKETS 16
+            PROPERTIES (
+                "replication_allocation" = "tag.location.default: 1",
+                "storage_format" = "V2"
+            );
+            """
+
+        // DDL/DML return 1 row and 3 column, the only value is update row count
+        assertTrue(result1.size() == 1)
+        assertTrue(result1[0].size() == 1)
+        assertTrue(result1[0][0] == 0, "Create table should update 0 rows")
+
+        // insert 1 row to check whether the table is ok
+        def result2 = sql """ INSERT INTO ${testTable} VALUES
+                        (1,2023-09-01,1,1,1)
+                        """
+        assertTrue(result2.size() == 1)
+        assertTrue(result2[0].size() == 1)
+        assertTrue(result2[0][0] == 1, "Insert should update 1 rows")
+    }
+
+    def load_from_hdfs_partition = {testTablex, label, hdfsFilePath, format, brokerName, hdfsUser, hdfsPasswd ->
+        def result1= sql """
+                        LOAD LABEL ${label} (
+                            DATA INFILE("${hdfsFilePath}")
+                            INTO TABLE ${testTablex}
+                            PARTITION(`p202309`)
+                            COLUMNS TERMINATED BY ","
+                            FORMAT as "${format}"
+                        )
+                        with BROKER "${brokerName}" (
+                        "username"="${hdfsUser}",
+                        "password"="${hdfsPasswd}")
+                        PROPERTIES  (
+                        "timeout"="1200",
+                        "max_filter_ratio"="0.1");
+                        """
+
+        assertTrue(result1.size() == 1)
+        assertTrue(result1[0].size() == 1)
+        assertTrue(result1[0][0] == 0, "Query OK, 0 rows affected")
+    }
+
+    def check_load_result = {checklabel, testTablex ->
+        max_try_milli_secs = 10000
+        while(max_try_milli_secs) {
+            result = sql "show load where label = '${checklabel}'"
+            log.info("result: ${result}")
+            if(result[0][2] == "FINISHED") {
+                //sql "sync"
+                qt_select "select * from ${testTablex} order by k1"
+                break
+            } else {
+                sleep(1000) // wait 1 second every time
+                max_try_milli_secs -= 1000
+                if(max_try_milli_secs <= 0) {
+                    assertEquals(1, 2)
+                }
+            }
+        }
+    }
+
+    // if 'enableHdfs' in regression-conf.groovy has been set to true,
+    // the test will run these case as below.
+    if (enableHdfs()) {
+        brokerName = getBrokerName()
+        hdfsUser = getHdfsUser()
+        hdfsPasswd = getHdfsPasswd()
+        def hdfs_csv_file_path = uploadToHdfs "load_p0/broker_load/broker_load_with_partition.csv"
+        //def hdfs_csv_file_path = "hdfs://ip:port/testfile"
+
+        try {
+            sql "DROP TABLE IF EXISTS ${testTable}"
+            create_test_table.call(testTable)
+
+            def test_load_label = UUID.randomUUID().toString().replaceAll("-", "")
+            load_from_hdfs_partition.call(testTable, test_load_label, hdfs_csv_file_path, "csv",
+                                brokerName, hdfsUser, hdfsPasswd)
+
+            check_load_result.call(test_load_label, testTable)
+        } finally {
+            try_sql("DROP TABLE IF EXISTS ${testTable}")
+        }
+    }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 17/24: [fix](mtmv)resolve task tvf concurrent modification exception #30497

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 9e63015e2fd56484b38bbfbfb2f75d17179dfead
Author: zhangdong <49...@qq.com>
AuthorDate: Tue Jan 30 11:55:20 2024 +0800

    [fix](mtmv)resolve task tvf concurrent modification exception #30497
    
    LinkedList is not thread safe and there may be issues with concurrent queries and writes. Change it to ConcurrentLinkedQueue
    
    Caused by: java.util.ConcurrentModificationException
            at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) ~[?:1.8.0_131]
            at java.util.LinkedList$ListItr.next(LinkedList.java:888) ~[?:1.8.0_131]
            at org.apache.doris.tablefunction.MetadataGenerator.taskMetadataResult(MetadataGenerator.java:694) ~[doris-fe.jar:1.2-SNAPSHOT]
            at org.apache.doris.tablefunction.MetadataGenerator.getMetadataTable(MetadataGenerator.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
            at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2195) ~[doris-fe.jar:1.2-SNAPSHOT]
            ... 13 more
---
 .../java/org/apache/doris/job/extensions/mtmv/MTMVJob.java    |  2 +-
 .../src/main/java/org/apache/doris/mtmv/MTMVJobInfo.java      | 11 +++++------
 2 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVJob.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVJob.java
index 678155170bd..4f44b2e14b9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVJob.java
@@ -185,7 +185,7 @@ public class MTMVJob extends AbstractJob<MTMVTask, MTMVTaskContext> {
             LOG.warn("get mtmv failed", e);
             return Lists.newArrayList();
         }
-        return mtmv.getJobInfo().getHistoryTasks();
+        return Lists.newArrayList(mtmv.getJobInfo().getHistoryTasks());
     }
 
     @Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobInfo.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobInfo.java
index aca04ee706f..b9a65e4d543 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVJobInfo.java
@@ -20,10 +20,9 @@ package org.apache.doris.mtmv;
 import org.apache.doris.common.Config;
 import org.apache.doris.job.extensions.mtmv.MTMVTask;
 
-import com.google.common.collect.Lists;
 import com.google.gson.annotations.SerializedName;
 
-import java.util.LinkedList;
+import java.util.concurrent.ConcurrentLinkedQueue;
 
 /**
  * MTMVJobInfo
@@ -32,11 +31,11 @@ public class MTMVJobInfo {
     @SerializedName("jobName")
     private String jobName;
     @SerializedName("ht")
-    private LinkedList<MTMVTask> historyTasks;
+    private ConcurrentLinkedQueue<MTMVTask> historyTasks;
 
     public MTMVJobInfo(String jobName) {
         this.jobName = jobName;
-        historyTasks = Lists.newLinkedList();
+        historyTasks = new ConcurrentLinkedQueue<>();
     }
 
     public String getJobName() {
@@ -49,11 +48,11 @@ public class MTMVJobInfo {
         }
         historyTasks.add(task);
         if (historyTasks.size() > Config.max_persistence_task_count) {
-            historyTasks.removeFirst();
+            historyTasks.poll();
         }
     }
 
-    public LinkedList<MTMVTask> getHistoryTasks() {
+    public ConcurrentLinkedQueue<MTMVTask> getHistoryTasks() {
         return historyTasks;
     }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 09/24: [Try_Fix](scan) try fix the scanner schedule logic to prevent excessive memory usage and timeout (#30515)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 129463f557c74eda05a618608ae3dea8d83ceede
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Mon Jan 29 23:41:55 2024 +0800

    [Try_Fix](scan) try fix the scanner schedule logic to prevent excessive memory usage and timeout (#30515)
---
 be/src/vec/exec/scan/pip_scanner_context.h | 14 ++++++++++++--
 be/src/vec/exec/scan/scanner_context.cpp   |  6 +++---
 be/src/vec/exec/scan/scanner_context.h     |  5 +++--
 3 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/be/src/vec/exec/scan/pip_scanner_context.h b/be/src/vec/exec/scan/pip_scanner_context.h
index c2490a5e0d0..484fe5b4003 100644
--- a/be/src/vec/exec/scan/pip_scanner_context.h
+++ b/be/src/vec/exec/scan/pip_scanner_context.h
@@ -40,7 +40,7 @@ public:
               _need_colocate_distribute(!_col_distribute_ids.empty()) {}
 
     Status get_block_from_queue(RuntimeState* state, vectorized::BlockUPtr* block, bool* eos,
-                                int id, bool wait = false) override {
+                                int id) override {
         {
             std::unique_lock l(_transfer_lock);
             if (state->is_cancelled()) {
@@ -97,6 +97,11 @@ public:
             (*block)->set_columns(std::move(m.mutable_columns()));
         }
 
+        // after return free blocks, should try to reschedule the scanner
+        if (should_be_scheduled()) {
+            this->reschedule_scanner_ctx();
+        }
+
         return Status::OK();
     }
 
@@ -284,7 +289,7 @@ public:
                                          limit_, max_bytes_in_blocks_queue, 1, local_state,
                                          dependency) {}
     Status get_block_from_queue(RuntimeState* state, vectorized::BlockUPtr* block, bool* eos,
-                                int id, bool wait = false) override {
+                                int id) override {
         if (_blocks_queue_buffered.empty()) {
             std::unique_lock l(_transfer_lock);
             if (state->is_cancelled()) {
@@ -349,6 +354,11 @@ public:
         }
         return_free_blocks();
 
+        // after return free blocks, should try to reschedule the scanner
+        if (should_be_scheduled()) {
+            this->reschedule_scanner_ctx();
+        }
+
         return Status::OK();
     }
 
diff --git a/be/src/vec/exec/scan/scanner_context.cpp b/be/src/vec/exec/scan/scanner_context.cpp
index 7691f159509..033709f950e 100644
--- a/be/src/vec/exec/scan/scanner_context.cpp
+++ b/be/src/vec/exec/scan/scanner_context.cpp
@@ -273,7 +273,7 @@ bool ScannerContext::empty_in_queue(int id) {
 }
 
 Status ScannerContext::get_block_from_queue(RuntimeState* state, vectorized::BlockUPtr* block,
-                                            bool* eos, int id, bool wait) {
+                                            bool* eos, int id) {
     std::vector<vectorized::BlockUPtr> merge_blocks;
     {
         std::unique_lock l(_transfer_lock);
@@ -295,9 +295,9 @@ Status ScannerContext::get_block_from_queue(RuntimeState* state, vectorized::Blo
         }
 
         // Wait for block from queue
-        if (wait) {
-            // scanner batch wait time
+        {
             SCOPED_TIMER(_scanner_wait_batch_timer);
+            // scanner batch wait time
             while (!(!_blocks_queue.empty() || done() || !status().ok() || state->is_cancelled())) {
                 if (!is_scheduled && _num_running_scanners == 0 && should_be_scheduled()) {
                     LOG(INFO) << debug_string();
diff --git a/be/src/vec/exec/scan/scanner_context.h b/be/src/vec/exec/scan/scanner_context.h
index 28aec83d6a2..59e4c45a52a 100644
--- a/be/src/vec/exec/scan/scanner_context.h
+++ b/be/src/vec/exec/scan/scanner_context.h
@@ -89,7 +89,7 @@ public:
     // Set eos to true if there is no more data to read.
     // And if eos is true, the block returned must be nullptr.
     virtual Status get_block_from_queue(RuntimeState* state, vectorized::BlockUPtr* block,
-                                        bool* eos, int id, bool wait = true);
+                                        bool* eos, int id);
 
     [[nodiscard]] Status validate_block_schema(Block* block);
 
@@ -134,7 +134,8 @@ public:
 
     // todo(wb) rethinking how to calculate ```_max_bytes_in_queue``` when executing shared scan
     inline bool should_be_scheduled() const {
-        return _cur_bytes_in_queue < _max_bytes_in_queue / 2;
+        return (_cur_bytes_in_queue < _max_bytes_in_queue / 2) &&
+               (_serving_blocks_num < allowed_blocks_num());
     }
 
     int get_available_thread_slot_num() {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 06/24: [regression test](stream load) test for `enable_profile` (#28534)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 5f20d7c5d006797791a18c640cd84a5f7c5cadd4
Author: Guangdong Liu <li...@gmail.com>
AuthorDate: Mon Jan 29 21:57:22 2024 +0800

    [regression test](stream load) test for `enable_profile` (#28534)
---
 .../data/load_p0/http_stream/test_http_stream.out  |  4 +++
 .../load_p0/http_stream/test_http_stream.groovy    | 39 ++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/regression-test/data/load_p0/http_stream/test_http_stream.out b/regression-test/data/load_p0/http_stream/test_http_stream.out
index 68fbf64f4bf..7ce24eea095 100644
--- a/regression-test/data/load_p0/http_stream/test_http_stream.out
+++ b/regression-test/data/load_p0/http_stream/test_http_stream.out
@@ -616,3 +616,7 @@
 1	test
 2	test
 
+-- !sql18 --
+1	test
+2	test
+
diff --git a/regression-test/suites/load_p0/http_stream/test_http_stream.groovy b/regression-test/suites/load_p0/http_stream/test_http_stream.groovy
index d6534432070..781732988e5 100644
--- a/regression-test/suites/load_p0/http_stream/test_http_stream.groovy
+++ b/regression-test/suites/load_p0/http_stream/test_http_stream.groovy
@@ -815,5 +815,44 @@ suite("test_http_stream", "p0") {
     } finally {
         try_sql "DROP TABLE IF EXISTS ${tableName17}"
     }
+
+    //  test enable_profile
+    def tableName18 = "test_http_stream_enable_profile"
+
+    try {
+        sql """
+       CREATE TABLE IF NOT EXISTS ${tableName18} (
+        `k1` int(11) NULL,
+        `k2` VARCHAR(20) NULL
+        ) ENGINE=OLAP
+        DUPLICATE KEY(`k1`)
+        DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+        PROPERTIES (
+        "replication_allocation" = "tag.location.default: 1"
+        );
+        """
+
+        streamLoad {
+            set 'version', '1'
+            set 'enable_profile','true'
+            set 'sql', """
+                    insert into ${db}.${tableName18} select c1, c2 from http_stream("format"="csv", "column_separator"="|", "trim_double_quotes"="true")
+                    """
+            time 10000
+            file '../stream_load/trim_double_quotes.csv'
+            check { result, exception, startTime, endTime ->
+                if (exception != null) {
+                    throw exception
+                }
+                log.info("http_stream result: ${result}".toString())
+                def json = parseJson(result)
+                assertEquals("success", json.Status.toLowerCase())
+            }
+        }
+
+        qt_sql18 "select * from ${tableName18} order by k1"
+    } finally {
+        try_sql "DROP TABLE IF EXISTS ${tableName18}"
+    }
 }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 10/24: [fix](move-memtable) check load timeout before close wait (#30526)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 28c4e6914911a3c8e09ee61e22de4bec82978c9b
Author: Kaijie Chen <ck...@apache.org>
AuthorDate: Tue Jan 30 09:44:03 2024 +0800

    [fix](move-memtable) check load timeout before close wait (#30526)
---
 be/src/vec/sink/writer/vtablet_writer_v2.cpp | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/be/src/vec/sink/writer/vtablet_writer_v2.cpp b/be/src/vec/sink/writer/vtablet_writer_v2.cpp
index bb4b60a3a86..f2fdbcaece3 100644
--- a/be/src/vec/sink/writer/vtablet_writer_v2.cpp
+++ b/be/src/vec/sink/writer/vtablet_writer_v2.cpp
@@ -553,6 +553,11 @@ Status VTabletWriterV2::close(Status exec_status) {
             SCOPED_TIMER(_close_load_timer);
             auto remain_ms = _state->execution_timeout() * 1000 -
                              _timeout_watch.elapsed_time() / 1000 / 1000;
+            if (remain_ms <= 0) {
+                LOG(WARNING) << "load timed out before close waiting, load_id="
+                             << print_id(_load_id);
+                return Status::TimedOut("load timed out before close waiting");
+            }
             for (const auto& [_, streams] : _streams_for_node) {
                 for (const auto& stream : streams->streams()) {
                     RETURN_IF_ERROR(stream->close_wait(remain_ms));


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 20/24: [improvement](thirdparty) introduce brpc stream patch to make stream write to socket in background bthread (#30458)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit b7c86d66faa5186316559a42a8965926b84d20ee
Author: HHoflittlefish777 <77...@users.noreply.github.com>
AuthorDate: Tue Jan 30 14:04:08 2024 +0800

    [improvement](thirdparty) introduce brpc stream patch to make stream write to socket in background bthread (#30458)
---
 .../patches/brpc-1.4.0-write-background.patch      | 170 +++++++++++++++++++++
 1 file changed, 170 insertions(+)

diff --git a/thirdparty/patches/brpc-1.4.0-write-background.patch b/thirdparty/patches/brpc-1.4.0-write-background.patch
new file mode 100644
index 00000000000..7c94df50654
--- /dev/null
+++ b/thirdparty/patches/brpc-1.4.0-write-background.patch
@@ -0,0 +1,170 @@
+diff --git a/src/brpc/controller.cpp b/src/brpc/controller.cpp
+index b6c8e750..c52e9451 100644
+--- a/src/brpc/controller.cpp
++++ b/src/brpc/controller.cpp
+@@ -1169,6 +1169,7 @@ void Controller::IssueRPC(int64_t start_realtime_us) {
+     wopt.pipelined_count = _pipelined_count;
+     wopt.auth_flags = _auth_flags;
+     wopt.ignore_eovercrowded = has_flag(FLAGS_IGNORE_EOVERCROWDED);
++    wopt.write_in_background = write_to_socket_in_background();
+     int rc;
+     size_t packet_size = 0;
+     if (user_packet_guard) {
+diff --git a/src/brpc/controller.h b/src/brpc/controller.h
+index 658cc695..9221f583 100644
+--- a/src/brpc/controller.h
++++ b/src/brpc/controller.h
+@@ -144,6 +144,7 @@ friend void policy::ProcessThriftRequest(InputMessageBase*);
+     static const uint32_t FLAGS_HEALTH_CHECK_CALL = (1 << 19);
+     static const uint32_t FLAGS_PB_SINGLE_REPEATED_TO_ARRAY = (1 << 20);
+     static const uint32_t FLAGS_MANAGE_HTTP_BODY_ON_ERROR = (1 << 21);
++    static const uint32_t FLAGS_WRITE_TO_SOCKET_IN_BACKGROUND = (1 << 22);
+ 
+ public:
+     struct Inheritable {
+@@ -350,6 +351,17 @@ public:
+     bool is_done_allowed_to_run_in_place() const
+     { return has_flag(FLAGS_ALLOW_DONE_TO_RUN_IN_PLACE); }
+ 
++    // Create a background KEEPWRITE bthread to write to socket when issuing
++    // RPCs, instead of trying to write to socket once in calling thread (see
++    // `Socket::StartWrite` in socket.cpp).
++    // The socket write could take some time (several microseconds maybe), if
++    // you cares about it and don't want the calling thread to be blocked, you
++    // can set this flag.
++    // Should provides better batch effect in situations like when you are
++    // continually issuing lots of async RPC calls in only one thread.
++    void set_write_to_socket_in_background(bool f) { set_flag(FLAGS_WRITE_TO_SOCKET_IN_BACKGROUND, f); }
++    bool write_to_socket_in_background() const { return has_flag(FLAGS_WRITE_TO_SOCKET_IN_BACKGROUND); }
++
+     // ------------------------------------------------------------------------
+     //                      Server-side methods.
+     // These calls shall be made from the server side only. Their results are
+diff --git a/src/brpc/socket.cpp b/src/brpc/socket.cpp
+index e3878c19..27748434 100644
+--- a/src/brpc/socket.cpp
++++ b/src/brpc/socket.cpp
+@@ -1620,7 +1620,7 @@ int Socket::StartWrite(WriteRequest* req, const WriteOptions& opt) {
+     // in some protocols(namely RTMP).
+     req->Setup(this);
+     
+-    if (ssl_state() != SSL_OFF) {
++    if (opt.write_in_background || ssl_state() != SSL_OFF) {
+         // Writing into SSL may block the current bthread, always write
+         // in the background.
+         goto KEEPWRITE_IN_BACKGROUND;
+diff --git a/src/brpc/socket.h b/src/brpc/socket.h
+index 6f710ee2..28a7ada6 100644
+--- a/src/brpc/socket.h
++++ b/src/brpc/socket.h
+@@ -269,10 +269,20 @@ public:
+         // Default: false
+         bool ignore_eovercrowded;
+ 
++        // The calling thread directly creates KeepWrite thread to write into
++        // this socket, skipping writing once.
++        // In situations like when you are continually issuing lots of
++        // StreamWrite or async RPC calls in only one thread, directly creating
++        // KeepWrite thread at first provides batch write effect and better
++        // performance. Otherwise, each write only writes one `msg` into socket
++        // and no KeepWrite thread can be created, which brings poor
++        // performance.
++        bool write_in_background;
++
+         WriteOptions()
+             : id_wait(INVALID_BTHREAD_ID), abstime(NULL)
+             , pipelined_count(0), auth_flags(0)
+-            , ignore_eovercrowded(false) {}
++            , ignore_eovercrowded(false), write_in_background(false) {}
+     };
+     int Write(butil::IOBuf *msg, const WriteOptions* options = NULL);
+ 
+diff --git a/src/brpc/stream.cpp b/src/brpc/stream.cpp
+index d8466d2a..2d565759 100644
+--- a/src/brpc/stream.cpp
++++ b/src/brpc/stream.cpp
+@@ -271,7 +271,8 @@ void Stream::TriggerOnConnectIfNeed() {
+     bthread_mutex_unlock(&_connect_mutex);
+ }
+ 
+-int Stream::AppendIfNotFull(const butil::IOBuf &data) {
++int Stream::AppendIfNotFull(const butil::IOBuf &data,
++                            const StreamWriteOptions* options) {
+     if (_cur_buf_size > 0) {
+         std::unique_lock<bthread_mutex_t> lck(_congestion_control_mutex);
+         if (_produced >= _remote_consumed + _cur_buf_size) {
+@@ -290,7 +291,9 @@ int Stream::AppendIfNotFull(const butil::IOBuf &data) {
+ 
+     size_t data_length = data.length();
+     butil::IOBuf copied_data(data);
+-    const int rc = _fake_socket_weak_ref->Write(&copied_data);
++    Socket::WriteOptions wopt;
++    wopt.write_in_background = options != NULL && options->write_in_background;
++    const int rc = _fake_socket_weak_ref->Write(&copied_data, &wopt);
+     if (rc != 0) {
+         // Stream may be closed by peer before
+         LOG(WARNING) << "Fail to write to _fake_socket, " << berror();
+@@ -679,13 +682,14 @@ void Stream::HandleRpcResponse(butil::IOBuf* response_buffer) {
+     policy::ProcessRpcResponse(msg);
+ }
+ 
+-int StreamWrite(StreamId stream_id, const butil::IOBuf &message) {
++int StreamWrite(StreamId stream_id, const butil::IOBuf &message,
++                const StreamWriteOptions* options) {
+     SocketUniquePtr ptr;
+     if (Socket::Address(stream_id, &ptr) != 0) {
+         return EINVAL;
+     }
+     Stream* s = (Stream*)ptr->conn();
+-    const int rc = s->AppendIfNotFull(message);
++    const int rc = s->AppendIfNotFull(message, options);
+     if (rc == 0) {
+         return 0;
+     }
+diff --git a/src/brpc/stream.h b/src/brpc/stream.h
+index fbf2d51d..410a5a09 100644
+--- a/src/brpc/stream.h
++++ b/src/brpc/stream.h
+@@ -82,6 +82,18 @@ struct StreamOptions {
+     StreamInputHandler* handler;
+ };
+ 
++struct StreamWriteOptions
++{
++    StreamWriteOptions() : write_in_background(false) {}
++
++    // Write message to socket in background thread.
++    // Provides batch write effect and better performance in situations when
++    // you are continually issuing lots of StreamWrite or async RPC calls in
++    // only one thread. Otherwise, each StreamWrite directly writes message into
++    // socket and brings poor performance.
++    bool write_in_background;
++};
++
+ // [Called at the client side]
+ // Create a stream at client-side along with the |cntl|, which will be connected
+ // when receiving the response with a stream from server-side. If |options| is
+@@ -104,7 +116,8 @@ int StreamAccept(StreamId* response_stream, Controller &cntl,
+ //  - EAGAIN: |stream_id| is created with positive |max_buf_size| and buf size
+ //            which the remote side hasn't consumed yet excceeds the number.
+ //  - EINVAL: |stream_id| is invalied or has been closed
+-int StreamWrite(StreamId stream_id, const butil::IOBuf &message);
++int StreamWrite(StreamId stream_id, const butil::IOBuf &message,
++                const StreamWriteOptions* options = NULL);
+ 
+ // Write util the pending buffer size is less than |max_buf_size| or orrur
+ // occurs
+diff --git a/src/brpc/stream_impl.h b/src/brpc/stream_impl.h
+index 259f0b77..f24b75a3 100644
+--- a/src/brpc/stream_impl.h
++++ b/src/brpc/stream_impl.h
+@@ -42,7 +42,8 @@ public:
+ 
+     // --------------------- SocketConnection --------------
+ 
+-    int AppendIfNotFull(const butil::IOBuf& msg);
++    int AppendIfNotFull(const butil::IOBuf& msg,
++                        const StreamWriteOptions* options = NULL);
+     static int Create(const StreamOptions& options,
+                       const StreamSettings *remote_settings,
+                       StreamId *id);
\ No newline at end of file


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 21/24: [fix](Nereids): fix wrong regression test (#30520)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 12827ceb16ced8d77027968f595ebb3ee3282c70
Author: jakevin <ja...@gmail.com>
AuthorDate: Tue Jan 30 14:26:25 2024 +0800

    [fix](Nereids): fix wrong regression test (#30520)
---
 .../push_down_top_n/push_down_top_n_distinct_through_join.out         | 2 +-
 .../push_down_top_n/push_down_top_n_distinct_through_join.groovy      | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out b/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
index 72452ed3c14..5c3ec397a99 100644
--- a/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
+++ b/regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
@@ -5,7 +5,7 @@ PhysicalResultSink
 ----PhysicalTopN[LOCAL_SORT]
 ------hashAgg[GLOBAL]
 --------hashAgg[LOCAL]
-----------NestedLoopJoin[LEFT_OUTER_JOIN](t1.id = t1.id)
+----------hashJoin[LEFT_OUTER_JOIN] hashCondition=((t1.id = t2.id)) otherCondition=()
 ------------PhysicalTopN[MERGE_SORT]
 --------------PhysicalTopN[LOCAL_SORT]
 ----------------hashAgg[LOCAL]
diff --git a/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy b/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
index 890e2ba897d..3ee3a916e7b 100644
--- a/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
+++ b/regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
@@ -51,11 +51,11 @@ suite("push_down_top_n_distinct_through_join") {
     """
 
     qt_push_down_topn_through_join """
-        explain shape plan select distinct * from (select t1.id from table_join t1 left join table_join t2 on t1.id = t1.id) t order by id limit 10;
+        explain shape plan select distinct * from (select t1.id from table_join t1 left join table_join t2 on t1.id = t2.id) t order by id limit 10;
     """
 
     qt_push_down_topn_through_join_data """
-        select distinct * from (select t1.id from table_join t1 left join table_join t2 on t1.id = t1.id) t order by id limit 10;
+        select distinct * from (select t1.id from table_join t1 left join table_join t2 on t1.id = t2.id) t order by id limit 10;
     """
 
     qt_push_down_topn_through_join """


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 14/24: [fix](Nereids): add order by when testing pkfk to avoid unstable res #30507

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 5731ed7aad1d86f52cc014fc19c14a474471873c
Author: 谢健 <ji...@gmail.com>
AuthorDate: Tue Jan 30 10:26:15 2024 +0800

    [fix](Nereids): add order by when testing pkfk to avoid unstable res #30507
---
 regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out    | 8 ++++----
 .../suites/nereids_rules_p0/pkfk/eliminate_inner.groovy           | 4 ++--
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out b/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out
index d1e060e7dd0..fbbbf32b065 100644
--- a/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out
+++ b/regression-test/data/nereids_rules_p0/pkfk/eliminate_inner.out
@@ -102,8 +102,8 @@ fk with limit
 
 -- !shape --
 PhysicalResultSink
---PhysicalLimit[GLOBAL]
-----PhysicalLimit[LOCAL]
+--PhysicalTopN[MERGE_SORT]
+----PhysicalTopN[LOCAL_SORT]
 ------PhysicalOlapScan[fkt_not_null]
 
 -- !res --
@@ -245,8 +245,8 @@ fk with limit
 -- !shape --
 PhysicalResultSink
 --filter(( not fk IS NULL))
-----PhysicalLimit[GLOBAL]
-------PhysicalLimit[LOCAL]
+----PhysicalTopN[MERGE_SORT]
+------PhysicalTopN[LOCAL_SORT]
 --------PhysicalOlapScan[fkt]
 
 -- !res --
diff --git a/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy b/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
index 6819c06d608..a2e750f3dd2 100644
--- a/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
+++ b/regression-test/suites/nereids_rules_p0/pkfk/eliminate_inner.groovy
@@ -96,7 +96,7 @@ suite("eliminate_inner") {
     check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join (select fkt_not_null1.fk from fkt_not_null as fkt_not_null1 group by fkt_not_null1.fk) fkt_not_null on pkt.pk = fkt_not_null.fk;", "with_pk_col")
     check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join (select * from fkt_not_null union select * from fkt_not_null) fkt_not_null on pkt.pk = fkt_not_null.fk;", "with_pk_col")
     check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join (select fk, ROW_NUMBER() OVER (PARTITION BY fk ORDER BY fk) AS RowNum from fkt_not_null) fkt_not_null on pkt.pk = fkt_not_null.fk;", "fk with window")
-    check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join (select fk from fkt_not_null limit 1) fkt_not_null on pkt.pk = fkt_not_null.fk;", "fk with limit")
+    check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join (select fk from fkt_not_null order by fk limit 1) fkt_not_null on pkt.pk = fkt_not_null.fk;", "fk with limit")
     check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join fkt_not_null on pkt.pk = fkt_not_null.fk where pkt.pk = 1 and fkt_not_null.fk = 1;", "pk with filter that same as fk")
     check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join fkt_not_null on pkt.pk = fkt_not_null.fk where pkt.pk = 1 and fkt_not_null.fk = 1 and fkt_not_null.f = 1;", "pk with filter that included same as fk")
     check_shape_res("select fkt_not_null.*, pkt.pk from pkt inner join fkt_not_null on pkt.pk = fkt_not_null.fk where pkt.p = 1 and fkt_not_null.fk = 1 and fkt_not_null.f = 1;;", "pk with filter that not same as fk")
@@ -109,7 +109,7 @@ suite("eliminate_inner") {
     check_shape_res("select fkt.*, pkt.pk from pkt inner join (select fkt1.fk from fkt as fkt1 group by fkt1.fk) fkt on pkt.pk = fkt.fk;", "with_pk_col")
     check_shape_res("select fkt.*, pkt.pk from pkt inner join (select * from fkt union select * from fkt) fkt on pkt.pk = fkt.fk;", "with_pk_col")
     check_shape_res("select fkt.*, pkt.pk from pkt inner join (select fk, ROW_NUMBER() OVER (PARTITION BY fk ORDER BY fk) AS RowNum from fkt) fkt on pkt.pk = fkt.fk;", "fk with window")
-    check_shape_res("select fkt.*, pkt.pk from pkt inner join (select fk from fkt limit 1) fkt on pkt.pk = fkt.fk;", "fk with limit")
+    check_shape_res("select fkt.*, pkt.pk from pkt inner join (select fk from fkt order by fk limit 1 ) fkt on pkt.pk = fkt.fk;", "fk with limit")
     check_shape_res("select fkt.*, pkt.pk from pkt inner join fkt on pkt.pk = fkt.fk where pkt.pk = 1 and fkt.fk = 1;", "pk with filter that same as fk")
     check_shape_res("select fkt.*, pkt.pk from pkt inner join fkt on pkt.pk = fkt.fk where pkt.pk = 1 and fkt.fk = 1 and fkt.f = 1;", "pk with filter that included same as fk")
     check_shape_res("select fkt.*, pkt.pk from pkt inner join fkt on pkt.pk = fkt.fk where pkt.p = 1 and fkt.fk = 1 and fkt.f = 1;;", "pk with filter that not same as fk")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 13/24: [improvement](index)add index_id column in show index stmt (#30431)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit b712f0b8102c66c3b8166c6752a5190bb6c5a867
Author: qiye <ji...@gmail.com>
AuthorDate: Tue Jan 30 10:17:20 2024 +0800

    [improvement](index)add index_id column in show index stmt (#30431)
---
 .../org/apache/doris/analysis/ShowIndexStmt.java   |  1 +
 .../java/org/apache/doris/qe/ShowExecutor.java     |  3 +-
 .../data/index_p0/test_bitmap_index.out            | 58 ------------------
 .../test_decimal_bitmap_index_multi_page.out       |  3 -
 .../data/inverted_index_p0/test_bitmap_index.out   | 40 -------------
 .../data/inverted_index_p0/test_inverted_index.out | 52 -----------------
 .../suites/index_p0/test_bitmap_index.groovy       | 68 ++++++++++++++++++++--
 .../test_decimal_bitmap_index_multi_page.groovy    |  5 +-
 .../inverted_index_p0/test_bitmap_index.groovy     | 46 ++++++++++++++-
 .../inverted_index_p0/test_inverted_index.groovy   | 58 +++++++++++++++++-
 10 files changed, 168 insertions(+), 166 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java
index b35f2e77287..b2b9e8fc58c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowIndexStmt.java
@@ -46,6 +46,7 @@ public class ShowIndexStmt extends ShowStmt {
                     .addColumn(new Column("Index_type", ScalarType.createVarchar(80)))
                     .addColumn(new Column("Comment", ScalarType.createVarchar(80)))
                     .addColumn(new Column("Properties", ScalarType.createVarchar(200)))
+                    .addColumn(new Column("Index_id", ScalarType.createVarchar(30)))
                     .build();
     private String dbName;
     private TableName tableName;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index f3dac9eed3d..dbe0a27f931 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -1082,7 +1082,8 @@ public class ShowExecutor {
                 for (Index index : indexes) {
                     rows.add(Lists.newArrayList(showStmt.getTableName().toString(), "", index.getIndexName(),
                             "", String.join(",", index.getColumns()), "", "", "", "",
-                            "", index.getIndexType().name(), index.getComment(), index.getPropertiesString()));
+                            "", index.getIndexType().name(), index.getComment(), index.getPropertiesString(),
+                            String.valueOf(index.getIndexId())));
                 }
             } finally {
                 table.readUnlock();
diff --git a/regression-test/data/index_p0/test_bitmap_index.out b/regression-test/data/index_p0/test_bitmap_index.out
index d076aabae97..1ee61ff37e9 100644
--- a/regression-test/data/index_p0/test_bitmap_index.out
+++ b/regression-test/data/index_p0/test_bitmap_index.out
@@ -16,23 +16,6 @@ k13	DATETIME	Yes	false	\N	NONE
 k14	DATETIME(3)	Yes	false	\N	NONE
 k15	DATETIME(6)	Yes	false	\N	NONE
 
--- !sql --
-regression_test_index_p0.test_bitmap_index_dup		index1		k1						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index2		k2						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index3		k3						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index4		k4						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index5		k5						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index6		k6						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index7		k7						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index8		k8						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index9		k9						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index10		k10						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index11		k11						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index12		k12						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index13		k13						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index14		k14						BITMAP		
-regression_test_index_p0.test_bitmap_index_dup		index15		k15						BITMAP		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	2022-05-31	2022-05-31T10:00	2022-05-31T10:00:00.111	2022-05-31T10:00:00.111111
 
@@ -54,23 +37,6 @@ k14	DATETIME(3)	Yes	true	\N
 k15	DATETIME(6)	Yes	true	\N	
 v1	INT	Yes	false	\N	SUM
 
--- !sql --
-regression_test_index_p0.test_bitmap_index_agg		index1		k1						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index2		k2						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index3		k3						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index4		k4						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index5		k5						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index6		k6						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index7		k7						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index8		k8						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index9		k9						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index10		k10						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index11		k11						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index12		k12						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index13		k13						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index14		k14						BITMAP		
-regression_test_index_p0.test_bitmap_index_agg		index15		k15						BITMAP		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	2022-05-31	2022-05-31T10:00	2022-05-31T10:00:00.111	2022-05-31T10:00:00.111111	1
 
@@ -92,24 +58,6 @@ k14	DATETIME(3)	Yes	false	\N	NONE
 k15	DATETIME(6)	Yes	false	\N	NONE
 v1	INT	Yes	false	\N	NONE
 
--- !sql --
-regression_test_index_p0.test_bitmap_index_unique		index1		k1						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index2		k2						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index3		k3						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index4		k4						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index5		k5						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index6		k6						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index7		k7						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index8		k8						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index9		k9						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index10		k10						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index11		k11						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index12		k12						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index13		k13						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index14		k14						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index15		k15						BITMAP		
-regression_test_index_p0.test_bitmap_index_unique		index16		v1						BITMAP		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	2022-05-31	2022-05-31T10:00	2022-05-31T10:00:00.111	2022-05-31T10:00:00.111111	1
 
@@ -121,9 +69,6 @@ block_version	INT	Yes	false	\N	REPLACE
 vehicle_mode	INT	Yes	false	\N	REPLACE
 usage_mode	INT	Yes	false	\N	REPLACE
 
--- !sql --
-regression_test_index_p0.test_bitmap_index_unique_mor_delete		vid_bitmap_index		vid						BITMAP		
-
 -- !sql --
 2
 
@@ -148,9 +93,6 @@ block_version	INT	Yes	false	\N	NONE
 vehicle_mode	INT	Yes	false	\N	NONE
 usage_mode	INT	Yes	false	\N	NONE
 
--- !sql --
-regression_test_index_p0.test_bitmap_index_unique_mow_delete		vid_bitmap_index		vid						BITMAP		
-
 -- !sql --
 2
 
diff --git a/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out b/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out
index 8e1edd8ea28..aeb0972b318 100644
--- a/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out
+++ b/regression-test/data/index_p0/test_decimal_bitmap_index_multi_page.out
@@ -2,9 +2,6 @@
 -- !sql --
 a	DECIMAL(12, 6)	No	true	\N	
 
--- !sql --
-regression_test_index_p0.test_decimal_bitmap_index_multi_page		bitmap_index_multi_page		a						BITMAP		
-
 -- !sql --
 0.000001
 1.000001
diff --git a/regression-test/data/inverted_index_p0/test_bitmap_index.out b/regression-test/data/inverted_index_p0/test_bitmap_index.out
index d13f8749427..f61defcbb68 100644
--- a/regression-test/data/inverted_index_p0/test_bitmap_index.out
+++ b/regression-test/data/inverted_index_p0/test_bitmap_index.out
@@ -12,19 +12,6 @@ k9	LARGEINT	Yes	false	\N	NONE
 k10	DECIMAL	Yes	false	\N	NONE
 k11	BOOLEAN	Yes	false	\N	NONE
 
--- !sql --
-regression_test_inverted_index_p0.test_bitmap_index_dup		index1		k1						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index2		k2						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index3		k3						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index4		k4						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index5		k5						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index6		k6						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index7		k7						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index8		k8						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index9		k9						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index10		k10						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_dup		index11		k11						BITMAP		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true
 
@@ -42,19 +29,6 @@ k10	DECIMAL	Yes	true	\N
 k11	BOOLEAN	Yes	true	\N	
 v1	INT	Yes	false	\N	SUM
 
--- !sql --
-regression_test_inverted_index_p0.test_bitmap_index_agg		index1		k1						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index2		k2						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index3		k3						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index4		k4						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index5		k5						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index6		k6						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index7		k7						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index8		k8						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index9		k9						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index10		k10						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_agg		index11		k11						BITMAP		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	1
 
@@ -72,20 +46,6 @@ k10	DECIMAL	Yes	true	\N
 k11	BOOLEAN	Yes	true	\N	
 v1	INT	Yes	false	\N	NONE
 
--- !sql --
-regression_test_inverted_index_p0.test_bitmap_index_unique		index1		k1						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index2		k2						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index3		k3						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index4		k4						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index5		k5						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index6		k6						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index7		k7						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index8		k8						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index9		k9						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index10		k10						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index11		k11						BITMAP		
-regression_test_inverted_index_p0.test_bitmap_index_unique		index12		v1						BITMAP		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	1
 
diff --git a/regression-test/data/inverted_index_p0/test_inverted_index.out b/regression-test/data/inverted_index_p0/test_inverted_index.out
index 2a673582ec4..0c11ea99116 100644
--- a/regression-test/data/inverted_index_p0/test_inverted_index.out
+++ b/regression-test/data/inverted_index_p0/test_inverted_index.out
@@ -16,23 +16,6 @@ k13	DATETIME	Yes	false	\N	NONE
 k14	DATETIME(3)	Yes	false	\N	NONE
 k15	DATETIME(6)	Yes	false	\N	NONE
 
--- !sql --
-regression_test_inverted_index_p0.test_inverted_index_dup		index1		k1						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index2		k2						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index3		k3						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index4		k4						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index5		k5						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index6		k6						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index7		k7						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index8		k8						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index9		k9						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index10		k10						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index11		k11						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index12		k12						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index13		k13						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index14		k14						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_dup		index15		k15						INVERTED		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	2022-05-31	2022-05-31T10:00	2022-05-31T10:00:00.111	2022-05-31T10:00:00.111111
 
@@ -54,23 +37,6 @@ k14	DATETIME(3)	Yes	true	\N
 k15	DATETIME(6)	Yes	true	\N	
 v1	INT	Yes	false	\N	SUM
 
--- !sql --
-regression_test_inverted_index_p0.test_inverted_index_agg		index1		k1						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index2		k2						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index3		k3						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index4		k4						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index5		k5						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index6		k6						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index7		k7						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index8		k8						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index9		k9						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index10		k10						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index11		k11						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index12		k12						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index13		k13						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index14		k14						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_agg		index15		k15						INVERTED		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	2022-05-31	2022-05-31T10:00	2022-05-31T10:00:00.111	2022-05-31T10:00:00.111111	1
 
@@ -92,24 +58,6 @@ k14	DATETIME(3)	Yes	false	\N	NONE
 k15	DATETIME(6)	Yes	false	\N	NONE
 v1	INT	Yes	false	\N	NONE
 
--- !sql --
-regression_test_inverted_index_p0.test_inverted_index_unique		index1		k1						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index2		k2						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index3		k3						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index4		k4						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index5		k5						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index6		k6						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index7		k7						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index8		k8						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index9		k9						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index10		k10						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index11		k11						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index12		k12						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index13		k13						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index14		k14						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index15		k15						INVERTED		
-regression_test_inverted_index_p0.test_inverted_index_unique		index16		v1						INVERTED		
-
 -- !sql --
 1	1	1	1	1	1	2022-05-31	2022-05-31T10:00	1	1	true	2022-05-31	2022-05-31T10:00	2022-05-31T10:00:00.111	2022-05-31T10:00:00.111111	1
 
diff --git a/regression-test/suites/index_p0/test_bitmap_index.groovy b/regression-test/suites/index_p0/test_bitmap_index.groovy
index f36b3b89717..db442f7fec2 100644
--- a/regression-test/suites/index_p0/test_bitmap_index.groovy
+++ b/regression-test/suites/index_p0/test_bitmap_index.groovy
@@ -79,7 +79,24 @@ suite("test_bitmap_index") {
 
     sql "insert into ${tbName1} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,'2022-05-31','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111');"
     qt_sql "desc ${tbName1};"
-    qt_sql "SHOW INDEX FROM ${tbName1};"
+    def show_result = sql "show index from ${tbName1}"
+    logger.info("show index from " + tbName1 + " result: " + show_result)
+    assertEquals(show_result.size(), 15)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
+    assertEquals(show_result[12][2], "index13")
+    assertEquals(show_result[13][2], "index14")
+    assertEquals(show_result[14][2], "index15")
     qt_sql "select * from ${tbName1};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName1};"
@@ -166,7 +183,24 @@ suite("test_bitmap_index") {
 
     sql "insert into ${tbName2} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,'2022-05-31','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111',1);"
     qt_sql "desc ${tbName2};"
-    qt_sql "SHOW INDEX FROM ${tbName2};"
+    show_result = sql "show index from ${tbName2}"
+    logger.info("show index from " + tbName2 + " result: " + show_result)
+    assertEquals(show_result.size(), 15)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
+    assertEquals(show_result[12][2], "index13")
+    assertEquals(show_result[13][2], "index14")
+    assertEquals(show_result[14][2], "index15")
     qt_sql "select * from ${tbName2};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName2};"
@@ -249,7 +283,25 @@ suite("test_bitmap_index") {
 
     sql "insert into ${tbName3} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,'2022-05-31','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111',1);"
     qt_sql "desc ${tbName3};"
-    qt_sql "SHOW INDEX FROM ${tbName3};"
+    show_result = sql "show index from ${tbName3}"
+    logger.info("show index from " + tbName3 + " result: " + show_result)
+    assertEquals(show_result.size(), 16)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
+    assertEquals(show_result[12][2], "index13")
+    assertEquals(show_result[13][2], "index14")
+    assertEquals(show_result[14][2], "index15")
+    assertEquals(show_result[15][2], "index16")
     qt_sql "select * from ${tbName3};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName3};"
@@ -319,7 +371,10 @@ suite("test_bitmap_index") {
         sql "insert into ${tbName4}(create_time,vid,report_time,block_version,vehicle_mode,usage_mode) values('2023-08-25 11:00:00','123','2023-08-25 11:00:00',2,2,2);"
         sql "insert into ${tbName4}(create_time,vid,report_time,block_version,vehicle_mode,usage_mode) values('2023-08-25 12:00:00','123','2023-08-25 12:00:00',3,3,3);"
         qt_sql "desc ${tbName4};"
-        qt_sql "SHOW INDEX FROM ${tbName4};"
+        show_result = sql "show index from ${tbName4}"
+        logger.info("show index from " + tbName4 + " result: " + show_result)
+        assertEquals(show_result.size(), 1)
+        assertEquals(show_result[0][2], "vid_bitmap_index")
         sql "delete from ${tbName4} where vid='123' and report_time='2023-08-25 12:00:00' and create_time='2023-08-25 12:00:00';"
         qt_sql "select count(*) from ${tbName4}; "
         qt_sql "select count(*) from ${tbName4} where vid='123'; "
@@ -394,7 +449,10 @@ suite("test_bitmap_index") {
         sql "insert into ${tbName5}(create_time,vid,report_time,block_version,vehicle_mode,usage_mode) values('2023-08-25 11:00:00','123','2023-08-25 11:00:00',2,2,2);"
         sql "insert into ${tbName5}(create_time,vid,report_time,block_version,vehicle_mode,usage_mode) values('2023-08-25 12:00:00','123','2023-08-25 12:00:00',3,3,3);"
         qt_sql "desc ${tbName5};"
-        qt_sql "SHOW INDEX FROM ${tbName5};"
+        show_result = sql "show index from ${tbName5}"
+        logger.info("show index from " + tbName5 + " result: " + show_result)
+        assertEquals(show_result.size(), 1)
+        assertEquals(show_result[0][2], "vid_bitmap_index")
         sql "delete from ${tbName5} where vid='123' and report_time='2023-08-25 12:00:00' and create_time='2023-08-25 12:00:00';"
         qt_sql "select count(*) from ${tbName5}; "
         qt_sql "select count(*) from ${tbName5} where vid='123'; "
diff --git a/regression-test/suites/index_p0/test_decimal_bitmap_index_multi_page.groovy b/regression-test/suites/index_p0/test_decimal_bitmap_index_multi_page.groovy
index 607c6a88995..88f44808e33 100644
--- a/regression-test/suites/index_p0/test_decimal_bitmap_index_multi_page.groovy
+++ b/regression-test/suites/index_p0/test_decimal_bitmap_index_multi_page.groovy
@@ -18,7 +18,10 @@ suite("test_decimal_bitmap_index_multi_page") {
     def tbName = "test_decimal_bitmap_index_multi_page"
 
     qt_sql "desc ${tbName};"
-    qt_sql "SHOW INDEX FROM ${tbName};"
+    def show_result = sql "show index from ${tbName}"
+    logger.info("show index from " + tbName + " result: " + show_result)
+    assertEquals(show_result.size(), 1)
+    assertEquals(show_result[0][2], "bitmap_index_multi_page")
     qt_sql "select * from ${tbName} order by a ASC limit 3;"
     qt_sql "select * from ${tbName} order by a DESC limit 3;"
 
diff --git a/regression-test/suites/inverted_index_p0/test_bitmap_index.groovy b/regression-test/suites/inverted_index_p0/test_bitmap_index.groovy
index 0b360f3f6bf..23944288cd7 100644
--- a/regression-test/suites/inverted_index_p0/test_bitmap_index.groovy
+++ b/regression-test/suites/inverted_index_p0/test_bitmap_index.groovy
@@ -71,7 +71,20 @@ suite("test_bitmap_index", "inverted_index") {
 
     sql "insert into ${tbName1} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1);"
     qt_sql "desc ${tbName1};"
-    qt_sql "SHOW INDEX FROM ${tbName1};"
+    def show_result = sql "show index from ${tbName1}"
+    logger.info("show index from " + tbName1 + " result: " + show_result)
+    assertEquals(show_result.size(), 11)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
     qt_sql "select * from ${tbName1};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName1};"
@@ -150,7 +163,20 @@ suite("test_bitmap_index", "inverted_index") {
 
     sql "insert into ${tbName2} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,1);"
     qt_sql "desc ${tbName2};"
-    qt_sql "SHOW INDEX FROM ${tbName2};"
+    show_result = sql "show index from ${tbName2}"
+    logger.info("show index from " + tbName2 + " result: " + show_result)
+    assertEquals(show_result.size(), 11)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
     qt_sql "select * from ${tbName2};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName2};"
@@ -225,7 +251,21 @@ suite("test_bitmap_index", "inverted_index") {
 
     sql "insert into ${tbName3} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,1);"
     qt_sql "desc ${tbName3};"
-    qt_sql "SHOW INDEX FROM ${tbName3};"
+    show_result = sql "show index from ${tbName3}"
+    logger.info("show index from " + tbName3 + " result: " + show_result)
+    assertEquals(show_result.size(), 12)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
     qt_sql "select * from ${tbName3};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName3};"
diff --git a/regression-test/suites/inverted_index_p0/test_inverted_index.groovy b/regression-test/suites/inverted_index_p0/test_inverted_index.groovy
index 25cfadff448..f4280c31163 100644
--- a/regression-test/suites/inverted_index_p0/test_inverted_index.groovy
+++ b/regression-test/suites/inverted_index_p0/test_inverted_index.groovy
@@ -79,7 +79,24 @@ suite("test_inverted_index", "inverted_index") {
 
     sql "insert into ${tbName1} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,'2022-05-31','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111');"
     qt_sql "desc ${tbName1};"
-    qt_sql "SHOW INDEX FROM ${tbName1};"
+    def show_result = sql "show index from ${tbName1}"
+    logger.info("show index from " + tbName1 + " result: " + show_result)
+    assertEquals(show_result.size(), 15)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
+    assertEquals(show_result[12][2], "index13")
+    assertEquals(show_result[13][2], "index14")
+    assertEquals(show_result[14][2], "index15")
     qt_sql "select * from ${tbName1};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName1};"
@@ -166,7 +183,24 @@ suite("test_inverted_index", "inverted_index") {
 
     sql "insert into ${tbName2} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,'2022-05-31','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111',1);"
     qt_sql "desc ${tbName2};"
-    qt_sql "SHOW INDEX FROM ${tbName2};"
+    show_result = sql "show index from ${tbName2}"
+    logger.info("show index from " + tbName2 + " result: " + show_result)
+    assertEquals(show_result.size(), 15)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
+    assertEquals(show_result[12][2], "index13")
+    assertEquals(show_result[13][2], "index14")
+    assertEquals(show_result[14][2], "index15")
     qt_sql "select * from ${tbName2};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName2};"
@@ -253,7 +287,25 @@ suite("test_inverted_index", "inverted_index") {
 
     sql "insert into ${tbName3} values(1,1,1,1,'1','1','2022-05-31','2022-05-31 10:00:00',1,1.0,1,'2022-05-31','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111','2022-05-31 10:00:00.111111',1);"
     qt_sql "desc ${tbName3};"
-    qt_sql "SHOW INDEX FROM ${tbName3};"
+    show_result = sql "show index from ${tbName3}"
+    logger.info("show index from " + tbName3 + " result: " + show_result)
+    assertEquals(show_result.size(), 16)
+    assertEquals(show_result[0][2], "index1")
+    assertEquals(show_result[1][2], "index2")
+    assertEquals(show_result[2][2], "index3")
+    assertEquals(show_result[3][2], "index4")
+    assertEquals(show_result[4][2], "index5")
+    assertEquals(show_result[5][2], "index6")
+    assertEquals(show_result[6][2], "index7")
+    assertEquals(show_result[7][2], "index8")
+    assertEquals(show_result[8][2], "index9")
+    assertEquals(show_result[9][2], "index10")
+    assertEquals(show_result[10][2], "index11")
+    assertEquals(show_result[11][2], "index12")
+    assertEquals(show_result[12][2], "index13")
+    assertEquals(show_result[13][2], "index14")
+    assertEquals(show_result[14][2], "index15")
+    assertEquals(show_result[15][2], "index16")
     qt_sql "select * from ${tbName3};"
 
     sql "DROP INDEX IF EXISTS index1 ON ${tbName3};"


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 08/24: [Fix](jdk17) support start FE with JDK17 (#29658) (#30485)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 589e83561c69679136250a009ed4035d73780d65
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Mon Jan 29 23:19:08 2024 +0800

    [Fix](jdk17) support start FE with JDK17 (#29658) (#30485)
    
    Issue Number: close #30484
    
    problem:
    gson will use Java's reflection mechanism to generate a default Adapter, but JDK17 is prohibited from visiting such an access.
    
    solution:
    gson has provided solutions since 2.9.1, which can bypass this problem: Add support for reflection access filter by Marcono1234 · Pull Request #1905 · google/gson
    
    We need to upgrade the gson version and use this solution
---
 conf/fe.conf                                                          | 2 +-
 fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java | 2 ++
 fe/pom.xml                                                            | 2 +-
 3 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/conf/fe.conf b/conf/fe.conf
index a8d30963890..4bae10ef224 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -39,7 +39,7 @@ JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:
 JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time -Dlog4j2.formatMsgNoLookups=true"
 
 # For jdk 16+, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_16="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time --add-opens java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.ref=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/ [...]
+JAVA_OPTS_FOR_JDK_16="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"
 
 ##
 ## the lowercase properties are read by main program.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
index 54c109b61a5..fb1b9d8b5ff 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
@@ -122,6 +122,7 @@ import com.google.gson.JsonObject;
 import com.google.gson.JsonParseException;
 import com.google.gson.JsonSerializationContext;
 import com.google.gson.JsonSerializer;
+import com.google.gson.ReflectionAccessFilter;
 import com.google.gson.TypeAdapter;
 import com.google.gson.TypeAdapterFactory;
 import com.google.gson.annotations.SerializedName;
@@ -282,6 +283,7 @@ public class GsonUtils {
     // Add any other adapters if necessary.
     private static final GsonBuilder GSON_BUILDER = new GsonBuilder().addSerializationExclusionStrategy(
                     new HiddenAnnotationExclusionStrategy()).enableComplexMapKeySerialization()
+            .addReflectionAccessFilter(ReflectionAccessFilter.BLOCK_INACCESSIBLE_JAVA)
             .registerTypeHierarchyAdapter(Table.class, new GuavaTableAdapter())
             .registerTypeHierarchyAdapter(Multimap.class, new GuavaMultimapAdapter())
             .registerTypeAdapterFactory(new PostProcessTypeAdapterFactory())
diff --git a/fe/pom.xml b/fe/pom.xml
index 33959e978d6..f64538fecba 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -239,7 +239,7 @@ under the License.
         <commons-pool.version>1.5.1</commons-pool.version>
         <commons-text.version>1.10.0</commons-text.version>
         <commons-validator.version>1.7</commons-validator.version>
-        <gson.version>2.8.9</gson.version>
+        <gson.version>2.10.1</gson.version>
         <guava.version>32.1.2-jre</guava.version>
         <jackson.version>2.15.2</jackson.version>
         <java-cup.version>0.11-a-czt02-cdh</java-cup.version>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 01/24: [fix](move-memtable) fix potential duplicate of TabletStream profile (#30397)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 49d17f2be2b5a36f31e353f39f93197f4b72e719
Author: Kaijie Chen <ck...@apache.org>
AuthorDate: Mon Jan 29 19:51:44 2024 +0800

    [fix](move-memtable) fix potential duplicate of TabletStream profile (#30397)
---
 be/src/runtime/load_stream.cpp       | 2 +-
 be/test/runtime/load_stream_test.cpp | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/be/src/runtime/load_stream.cpp b/be/src/runtime/load_stream.cpp
index c62f5e9da50..430abe609bf 100644
--- a/be/src/runtime/load_stream.cpp
+++ b/be/src/runtime/load_stream.cpp
@@ -292,8 +292,8 @@ Status IndexStream::_init_tablet_stream(TabletStreamSharedPtr& tablet_stream, in
                                         int64_t partition_id) {
     tablet_stream = std::make_shared<TabletStream>(_load_id, tablet_id, _txn_id, _load_stream_mgr,
                                                    _profile);
-    RETURN_IF_ERROR(tablet_stream->init(_schema, _id, partition_id));
     _tablet_streams_map[tablet_id] = tablet_stream;
+    RETURN_IF_ERROR(tablet_stream->init(_schema, _id, partition_id));
     return Status::OK();
 }
 
diff --git a/be/test/runtime/load_stream_test.cpp b/be/test/runtime/load_stream_test.cpp
index ca065f91a68..d248e1019f4 100644
--- a/be/test/runtime/load_stream_test.cpp
+++ b/be/test/runtime/load_stream_test.cpp
@@ -804,7 +804,8 @@ TEST_F(LoadStreamMgrTest, one_client_abnormal_tablet) {
     wait_for_ack(3);
     EXPECT_EQ(g_response_stat.num, 3);
     EXPECT_EQ(g_response_stat.success_tablet_ids.size(), 0);
-    EXPECT_EQ(g_response_stat.failed_tablet_ids.size(), 1);
+    EXPECT_EQ(g_response_stat.failed_tablet_ids.size(), 2);
+    EXPECT_EQ(g_response_stat.failed_tablet_ids[1], ABNORMAL_TABLET_ID);
 
     // server will close stream on CLOSE_LOAD
     wait_for_close();


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 23/24: [bugfix](paimon)add class loader (#30483)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 11f7b36dab86ec40d1695c4be1891cb82c499821
Author: wuwenchi <wu...@hotmail.com>
AuthorDate: Tue Jan 30 15:07:42 2024 +0800

    [bugfix](paimon)add class loader (#30483)
---
 .../org/apache/doris/paimon/PaimonJniScanner.java    | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java b/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
index ac497776004..fce74a27dc6 100644
--- a/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
+++ b/fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java
@@ -60,8 +60,10 @@ public class PaimonJniScanner extends JniScanner {
     private long tblId;
     private long lastUpdateTime;
     private RecordReader.RecordIterator<InternalRow> recordIterator = null;
+    private final ClassLoader classLoader;
 
     public PaimonJniScanner(int batchSize, Map<String, String> params) {
+        this.classLoader = this.getClass().getClassLoader();
         LOG.debug("params:{}", params);
         this.params = params;
         String[] requiredFields = params.get("required_fields").split(",");
@@ -87,9 +89,19 @@ public class PaimonJniScanner extends JniScanner {
 
     @Override
     public void open() throws IOException {
-        initTable();
-        initReader();
-        resetDatetimeV2Precision();
+        try {
+            // When the user does not specify hive-site.xml, Paimon will look for the file from the classpath:
+            //    org.apache.paimon.hive.HiveCatalog.createHiveConf:
+            //        `Thread.currentThread().getContextClassLoader().getResource(HIVE_SITE_FILE)`
+            // so we need to provide a classloader, otherwise it will cause NPE.
+            Thread.currentThread().setContextClassLoader(classLoader);
+            initTable();
+            initReader();
+            resetDatetimeV2Precision();
+        } catch (Exception e) {
+            LOG.warn("Failed to open paimon_scanner: " + e.getMessage(), e);
+            throw e;
+        }
     }
 
     private void initReader() throws IOException {
@@ -182,7 +194,7 @@ public class PaimonJniScanner extends JniScanner {
         PaimonTableCacheKey key = new PaimonTableCacheKey(ctlId, dbId, tblId, paimonOptionParams, dbName, tblName);
         TableExt tableExt = PaimonTableCache.getTable(key);
         if (tableExt.getCreateTime() < lastUpdateTime) {
-            LOG.warn("invalidate cacha table:{}, localTime:{}, remoteTime:{}", key, tableExt.getCreateTime(),
+            LOG.warn("invalidate cache table:{}, localTime:{}, remoteTime:{}", key, tableExt.getCreateTime(),
                     lastUpdateTime);
             PaimonTableCache.invalidateTableCache(key);
             tableExt = PaimonTableCache.getTable(key);


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 22/24: [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30541)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit ea427e8c51f126729dc8b45580455e00a49eebc0
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Tue Jan 30 15:04:20 2024 +0800

    [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30541)
    
    * [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO  table : InaccessibleObjectException (#30003)
---
 bin/start_be.sh                          | 7 ++++++-
 bin/start_fe.sh                          | 8 ++++----
 conf/be.conf                             | 3 +++
 conf/fe.conf                             | 4 ++--
 regression-test/pipeline/p0/conf/be.conf | 3 +++
 regression-test/pipeline/p0/conf/fe.conf | 3 +++
 6 files changed, 21 insertions(+), 7 deletions(-)

diff --git a/bin/start_be.sh b/bin/start_be.sh
index 7aff228c56b..34b2dbeaf87 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -298,7 +298,12 @@ CUR_DATE=$(date +%Y%m%d-%H%M%S)
 LOG_PATH="-DlogPath=${DORIS_HOME}/log/jni.log"
 COMMON_OPTS="-Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
 
-if [[ "${java_version}" -gt 8 ]]; then
+if [[ "${java_version}" -gt 16 ]]; then
+    if [[ -z ${JAVA_OPTS_FOR_JDK_17} ]]; then
+        JAVA_OPTS_FOR_JDK_17="-Xmx1024m ${LOG_PATH} -Xlog:gc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} ${COMMON_OPTS} --add-opens=java.base/java.net=ALL-UNNAMED"
+    fi
+    final_java_opt="${JAVA_OPTS_FOR_JDK_17}"
+elif [[ "${java_version}" -gt 8 ]]; then
     if [[ -z ${JAVA_OPTS_FOR_JDK_9} ]]; then
         JAVA_OPTS_FOR_JDK_9="-Xmx1024m ${LOG_PATH} -Xlog:gc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} ${COMMON_OPTS}"
     fi
diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index f52e47c93ed..f22138ce98a 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -181,12 +181,12 @@ java_version="$(
     jdk_version "${JAVA}"
 )"
 final_java_opt="${JAVA_OPTS}"
-if [[ "${java_version}" -ge 16 ]]; then
-    if [[ -z "${JAVA_OPTS_FOR_JDK_16}" ]]; then
-        echo "JAVA_OPTS_FOR_JDK_16 is not set in fe.conf" >>"${LOG_DIR}/fe.out"
+if [[ "${java_version}" -gt 16 ]]; then
+    if [[ -z "${JAVA_OPTS_FOR_JDK_17}" ]]; then
+        echo "JAVA_OPTS_FOR_JDK_17 is not set in fe.conf" >>"${LOG_DIR}/fe.out"
         exit 1
     fi
-    final_java_opt="${JAVA_OPTS_FOR_JDK_16}"
+    final_java_opt="${JAVA_OPTS_FOR_JDK_17}"
 elif [[ "${java_version}" -gt 8 ]]; then
     if [[ -z "${JAVA_OPTS_FOR_JDK_9}" ]]; then
         echo "JAVA_OPTS_FOR_JDK_9 is not set in fe.conf" >>"${LOG_DIR}/fe.out"
diff --git a/conf/be.conf b/conf/be.conf
index 9b1fa1d47c2..6ceec00a421 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -24,6 +24,9 @@ JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/b
 # For jdk 9+, this JAVA_OPTS will be used as default JVM options
 JAVA_OPTS_FOR_JDK_9="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
 
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives --add-opens=java.base/java.net=ALL-UNNAMED"
+
 # since 1.2, the JAVA_HOME need to be set to run BE process.
 # JAVA_HOME=/path/to/jdk/
 
diff --git a/conf/fe.conf b/conf/fe.conf
index 4bae10ef224..66794ddf3fa 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -38,8 +38,8 @@ JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:
 # For jdk 9+, this JAVA_OPTS_FOR_JDK_9 will be used as default G1 JVM options
 JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time -Dlog4j2.formatMsgNoLookups=true"
 
-# For jdk 16+, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_16="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"
 
 ##
 ## the lowercase properties are read by main program.
diff --git a/regression-test/pipeline/p0/conf/be.conf b/regression-test/pipeline/p0/conf/be.conf
index 3e74bf07c87..3bf1dd0980d 100644
--- a/regression-test/pipeline/p0/conf/be.conf
+++ b/regression-test/pipeline/p0/conf/be.conf
@@ -17,6 +17,9 @@
 
 PPROF_TMPDIR="$DORIS_HOME/log/"
 
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives --add-opens=java.base/java.net=ALL-UNNAMED"
+
 # INFO, WARNING, ERROR, FATAL
 sys_log_level = INFO
 
diff --git a/regression-test/pipeline/p0/conf/fe.conf b/regression-test/pipeline/p0/conf/fe.conf
index 0dea0f0286d..655aac0ba2f 100644
--- a/regression-test/pipeline/p0/conf/fe.conf
+++ b/regression-test/pipeline/p0/conf/fe.conf
@@ -33,6 +33,9 @@ JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx4096m -XX:
 # JAVA_OPTS_FOR_JDK_9="-Xmx4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/fe.jmap -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"
 JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/fe.jmap -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$DORIS_HOME/log/fe.gc.log.$CUR_DATE -Dlog4j2.formatMsgNoLookups=true"
 
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"
+
 ##
 ## the lowercase properties are read by main program.
 ##


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 04/24: [regression test](schema change) add case for column type change (#30472)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 57a8c75ddccc239b92d359f1eb40f4888213a04a
Author: Guangdong Liu <li...@gmail.com>
AuthorDate: Mon Jan 29 21:54:17 2024 +0800

    [regression test](schema change) add case for column type change (#30472)
---
 .../test_schema_change_agg_check_all_types.out     |  27 ++++
 .../test_schema_change_agg_check_all_types.groovy  | 170 ++++++++-------------
 2 files changed, 88 insertions(+), 109 deletions(-)

diff --git a/regression-test/data/schema_change_p0/test_schema_change_agg_check_all_types.out b/regression-test/data/schema_change_p0/test_schema_change_agg_check_all_types.out
index 3f7a29b7cd5..6c309341860 100644
--- a/regression-test/data/schema_change_p0/test_schema_change_agg_check_all_types.out
+++ b/regression-test/data/schema_change_p0/test_schema_change_agg_check_all_types.out
@@ -32,3 +32,30 @@
 10005	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	1.1100000143051147
 10006	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	1.11
 
+-- !add_float --
+10001	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10002	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10003	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10004	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10005	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	1.1100000143051147
+10006	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	1.11
+10007	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	1.11	1.21
+
+-- !float_to_varchar --
+10001	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0	0.0
+10002	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0	0.0
+10003	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0	0.0
+10004	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0	0.0
+10005	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0	1.1100000143051147
+10006	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0	1.11
+10007	2	3	8	10	13.2	3.4	9	a	b	c	2021-10-30	2021-10-30T00:00	1.11	1.21
+
+-- !varchar_to_float --
+10001	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10002	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10003	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10004	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	0.0
+10005	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	1.1100000143051147
+10006	2	3	4	5	6.6	1.7	9	a	b	c	2021-10-30	2021-10-30T00:00	0.0	1.11
+10007	2	3	8	10	13.2	3.4	9	a	b	c	2021-10-30	2021-10-30T00:00	1.11	1.21
+
diff --git a/regression-test/suites/schema_change_p0/test_schema_change_agg_check_all_types.groovy b/regression-test/suites/schema_change_p0/test_schema_change_agg_check_all_types.groovy
index a23c9094163..8eb70bb4ddf 100644
--- a/regression-test/suites/schema_change_p0/test_schema_change_agg_check_all_types.groovy
+++ b/regression-test/suites/schema_change_p0/test_schema_change_agg_check_all_types.groovy
@@ -18,16 +18,6 @@
 suite("test_schema_change_agg_check_all_types", "p0") {
     def tableName3 = "test_schema_change_agg_check_all_types"
 
-    def getJobState = { tableName ->
-        def jobStateResult = sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
-        return jobStateResult[0][9]
-    }
-
-    def getCreateViewState = { tableName ->
-        def createViewStateResult = sql """ SHOW ALTER TABLE MATERIALIZED VIEW WHERE IndexName='${tableName}' ORDER BY createtime DESC LIMIT 1 """
-        return createViewStateResult[0][8]
-    }
-
     sql """ DROP TABLE IF EXISTS ${tableName3} """
 
     sql """
@@ -55,22 +45,12 @@ suite("test_schema_change_agg_check_all_types", "p0") {
 
     // tinyint to smallint
     sql """ alter table ${tableName3} modify column k2 smallint key NULL"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
+
     sql """ insert into ${tableName3} values (10001, 2, 3, 4, 5, 6.6, 1.7, 8.8,
     'a', 'b', 'c', '2021-10-30', '2021-10-30 00:00:00') """
 
@@ -79,20 +59,9 @@ suite("test_schema_change_agg_check_all_types", "p0") {
 
     // smallint to int
     sql """ alter table ${tableName3} modify column k2 int key NULL"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
     sql """ insert into ${tableName3} values (10002, 2, 3, 4, 5, 6.6, 1.7, 8.8,
@@ -102,20 +71,10 @@ suite("test_schema_change_agg_check_all_types", "p0") {
 
     // int to bigint
     sql """ alter table ${tableName3} modify column k2 bigint key NULL"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
     sql """ insert into ${tableName3} values (10003, 2, 3, 4, 5, 6.6, 1.7, 8.8,
@@ -124,22 +83,11 @@ suite("test_schema_change_agg_check_all_types", "p0") {
     qt_int_to_bigint """ select * from ${tableName3} """
 
     // bigint to largeint
-
     sql """ alter table ${tableName3} modify column k2 largeint key NULL"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
     sql """ insert into ${tableName3} values (10004, 2, 3, 4, 5, 6.6, 1.7, 8.8,
@@ -150,37 +98,15 @@ suite("test_schema_change_agg_check_all_types", "p0") {
     // largeint to float
     sql """ alter table ${tableName3} add column k14 largeint replace not null default "0" after k13"""
 
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
     sql """ alter table ${tableName3} modify column k14 float replace not null default "0" after k13"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
     sql """ insert into ${tableName3} values (10005, 2, 3, 4, 5, 6.6, 1.7, 8.8,
@@ -190,20 +116,9 @@ suite("test_schema_change_agg_check_all_types", "p0") {
 
     // float to double
     sql """ alter table ${tableName3} modify column k14 double replace not null default "0" after k13"""
-    sleep(10)
-    max_try_num = 60
-    while (max_try_num--) {
-        String res = getJobState(tableName3)
-        if (res == "FINISHED" || res == "CANCELLED") {
-            assertEquals("FINISHED", res)
-            sleep(3000)
-            break
-        } else {
-            sleep(100)
-            if (max_try_num < 1){
-                assertEquals(1,2)
-            }
-        }
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
     }
 
     sql """ insert into ${tableName3} values (10006, 2, 3, 4, 5, 6.6, 1.7, 8.8,
@@ -211,5 +126,42 @@ suite("test_schema_change_agg_check_all_types", "p0") {
 
     qt_float_to_double """ select * from ${tableName3} """
 
+
+    // float to decimal & string
+    sql """ alter table ${tableName3} add column float1 float replace not null default "0" after k13"""
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
+    }
+    sql """ insert into ${tableName3} values (10007, 2, 3, 4, 5, 6.6, 1.7, 8.8,
+    'a', 'b', 'c', '2021-10-30', '2021-10-30 00:00:00',1.11,1.21) """
+
+    qt_add_float""" select * from ${tableName3} """
+
+    sql """ alter table ${tableName3} modify column float1 varchar replace not null default "0" after k13"""
+
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
+    }
+    sql """ insert into ${tableName3} values (10007, 2, 3, 4, 5, 6.6, 1.7, 8.8,
+    'a', 'b', 'c', '2021-10-30', '2021-10-30 00:00:00',1.11,'1.21') """
+    qt_float_to_varchar""" select * from ${tableName3} """
+
+
+    sql """ alter table ${tableName3} modify column float1 float replace not null default "0" after k13"""
+    waitForSchemaChangeDone {
+        sql """ SHOW ALTER TABLE COLUMN WHERE IndexName='${tableName3}' ORDER BY createtime DESC LIMIT 1 """
+        time 60
+    }
+
+    qt_varchar_to_float""" select * from ${tableName3} """
+
+
+    test {
+        sql """ alter table ${tableName3} modify column float1 decimal(9, 2) replace not null default "0" after k13"""
+        exception "Can not change FLOAT to DECIMAL32"
+    }
+
 }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 11/24: [enhancement](log) print detail error for segment compaction failure (#30503)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 0932dadcff7a1c12bb3e72d6c8668dcad6706b73
Author: zhannngchen <48...@users.noreply.github.com>
AuthorDate: Tue Jan 30 09:48:51 2024 +0800

    [enhancement](log) print detail error for segment compaction failure (#30503)
---
 be/src/olap/rowset/beta_rowset_writer.cpp | 11 ++++++++---
 be/src/olap/rowset/segcompaction.cpp      |  3 ++-
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/be/src/olap/rowset/beta_rowset_writer.cpp b/be/src/olap/rowset/beta_rowset_writer.cpp
index 04980816a5f..acc77b730ff 100644
--- a/be/src/olap/rowset/beta_rowset_writer.cpp
+++ b/be/src/olap/rowset/beta_rowset_writer.cpp
@@ -397,7 +397,8 @@ Status BetaRowsetWriter::_segcompaction_if_necessary() {
     }
     if (_segcompaction_status.load() != OK) {
         status = Status::Error<SEGCOMPACTION_FAILED>(
-                "BetaRowsetWriter::_segcompaction_if_necessary meet invalid state");
+                "BetaRowsetWriter::_segcompaction_if_necessary meet invalid state, error code: {}",
+                _segcompaction_status.load());
     } else if ((_num_segment - _segcompacted_point) >= config::segcompaction_batch_size) {
         SegCompactionCandidatesSharedPtr segments;
         status = _find_longest_consecutive_small_segment(segments);
@@ -426,7 +427,9 @@ Status BetaRowsetWriter::_segcompaction_rename_last_segments() {
     }
     if (_segcompaction_status.load() != OK) {
         return Status::Error<SEGCOMPACTION_FAILED>(
-                "BetaRowsetWriter::_segcompaction_rename_last_segments meet invalid state");
+                "BetaRowsetWriter::_segcompaction_rename_last_segments meet invalid state, error "
+                "code: {}",
+                _segcompaction_status.load());
     }
     if (!_is_segcompacted() || _segcompacted_point == _num_segment) {
         // no need if never segcompact before or all segcompacted
@@ -502,7 +505,9 @@ Status BetaRowsetWriter::_wait_flying_segcompaction() {
         LOG(INFO) << "wait flying segcompaction finish time:" << elapsed << "us";
     }
     if (_segcompaction_status.load() != OK) {
-        return Status::Error<SEGCOMPACTION_FAILED>("BetaRowsetWriter meet invalid state.");
+        return Status::Error<SEGCOMPACTION_FAILED>(
+                "BetaRowsetWriter meet invalid state, error code: {}",
+                _segcompaction_status.load());
     }
     return Status::OK();
 }
diff --git a/be/src/olap/rowset/segcompaction.cpp b/be/src/olap/rowset/segcompaction.cpp
index ad3e78a6081..32ade4392b3 100644
--- a/be/src/olap/rowset/segcompaction.cpp
+++ b/be/src/olap/rowset/segcompaction.cpp
@@ -242,7 +242,8 @@ Status SegcompactionWorker::_do_compact_segments(SegCompactionCandidatesSharedPt
         auto s = _get_segcompaction_reader(segments, tablet, schema, &reader_stats, row_sources_buf,
                                            is_key, column_ids, &reader);
         if (UNLIKELY(reader == nullptr || !s.ok())) {
-            return Status::Error<SEGCOMPACTION_INIT_READER>("failed to get segcompaction reader.");
+            return Status::Error<SEGCOMPACTION_INIT_READER>(
+                    "failed to get segcompaction reader. err: {}", s.to_string());
         }
 
         Merger::Statistics merger_stats;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 15/24: [bugfix](iceberg)fix read NULL with date partition (#30478)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 4648902350ea07cbf642a761d5d224afb162635c
Author: wuwenchi <wu...@hotmail.com>
AuthorDate: Tue Jan 30 10:30:57 2024 +0800

    [bugfix](iceberg)fix read NULL with date partition (#30478)
    
    * fix date
    
    * fix date
    
    * add case
---
 .../planner/external/iceberg/IcebergScanNode.java  | 19 +++++++++++--
 .../test_external_catalog_iceberg_partition.out    | 33 ++++++++++++++++++++++
 .../test_external_catalog_iceberg_partition.groovy | 21 ++++++++++++++
 3 files changed, 71 insertions(+), 2 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java
index ae3efccadc5..a335ccfa021 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java
@@ -69,6 +69,9 @@ import org.apache.iceberg.exceptions.NotFoundException;
 import org.apache.iceberg.expressions.Expression;
 import org.apache.iceberg.io.CloseableIterable;
 import org.apache.iceberg.types.Conversions;
+import org.apache.iceberg.types.Type;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.DateTimeUtil;
 import org.apache.iceberg.util.TableScanUtil;
 
 import java.io.IOException;
@@ -220,11 +223,23 @@ public class IcebergScanNode extends FileQueryScanNode {
                 List<String> partitionValues = new ArrayList<>();
                 if (isPartitionedTable) {
                     StructLike structLike = splitTask.file().partition();
+                    List<PartitionField> fields = splitTask.spec().fields();
+                    Types.StructType structType = icebergTable.schema().asStruct();
 
                     // set partitionValue for this IcebergSplit
                     for (int i = 0; i < structLike.size(); i++) {
-                        String partition = String.valueOf(structLike.get(i, Object.class));
-                        partitionValues.add(partition);
+                        Object obj = structLike.get(i, Object.class);
+                        String value = String.valueOf(obj);
+                        PartitionField partitionField = fields.get(i);
+                        if (partitionField.transform().isIdentity()) {
+                            Type type = structType.fieldType(partitionField.name());
+                            if (type != null && type.typeId().equals(Type.TypeID.DATE)) {
+                                // iceberg use integer to store date,
+                                // we need transform it to string
+                                value = DateTimeUtil.daysToIsoDate((Integer) obj);
+                            }
+                        }
+                        partitionValues.add(value);
                     }
 
                     // Counts the number of partitions read
diff --git a/regression-test/data/external_table_p2/iceberg/test_external_catalog_iceberg_partition.out b/regression-test/data/external_table_p2/iceberg/test_external_catalog_iceberg_partition.out
index c2582691ccb..a0fd39be5f2 100644
--- a/regression-test/data/external_table_p2/iceberg/test_external_catalog_iceberg_partition.out
+++ b/regression-test/data/external_table_p2/iceberg/test_external_catalog_iceberg_partition.out
@@ -71,3 +71,36 @@
 -- !q09 --
 100	0.3	test3	2023-01-03T00:00
 
+-- !q01 --
+1	true	2020-01-02	2020-01-02T01:02:03.123456
+
+-- !q02 --
+1	true	2020-01-02	2020-01-02T01:02:03.123456
+
+-- !q03 --
+1	true	2020-01-02	2020-01-02T01:02:03.123456
+
+-- !q04 --
+1	true	2020-01-02	2020-01-02T01:02:03.123456
+
+-- !q05 --
+1	true	2020-01-02	2020-01-02T01:02:03.123456
+
+-- !q06 --
+1	2020-01-02
+
+-- !q07 --
+1	2020-01-02
+
+-- !q08 --
+1	2020-01-02
+
+-- !q09 --
+1	2020-01-02
+
+-- !q10 --
+1	2020-01-02
+
+-- !q11 --
+1	2020-01-02
+
diff --git a/regression-test/suites/external_table_p2/iceberg/test_external_catalog_iceberg_partition.groovy b/regression-test/suites/external_table_p2/iceberg/test_external_catalog_iceberg_partition.groovy
index 9429887e8c0..dfdd923bcc4 100644
--- a/regression-test/suites/external_table_p2/iceberg/test_external_catalog_iceberg_partition.groovy
+++ b/regression-test/suites/external_table_p2/iceberg/test_external_catalog_iceberg_partition.groovy
@@ -55,9 +55,30 @@ suite("test_external_catalog_iceberg_partition", "p2,external,iceberg,external_r
             qt_q08 """ select * from iceberg_catalog.orc_partitioned_truncate_and_fields where t_int is null order by t_float """
             qt_q09 """ select * from iceberg_catalog.orc_partitioned_truncate_and_fields where t_int is not null order by t_float """
         }
+
+        // test date for partition and predict
+        def q01_date = {
+
+            qt_q01 """ select * from user_case_date_without_partition where d = '2020-01-02' """
+            qt_q02 """ select * from user_case_date_without_partition where d > '2020-01-01' """
+            qt_q03 """ select * from user_case_date_without_partition where d < '2020-01-03' """
+            qt_q04 """ select * from user_case_date_without_partition where ts < '2020-01-03' """
+            qt_q05 """ select * from user_case_date_without_partition where ts > '2020-01-01' """
+
+            qt_q06 """ select * from user_case_date_with_date_partition where d = '2020-01-02' """
+            qt_q07 """ select * from user_case_date_with_date_partition where d < '2020-01-03' """
+            qt_q08 """ select * from user_case_date_with_date_partition where d > '2020-01-01' """
+
+            qt_q09 """ select * from user_case_date_with_days_date_partition where d = '2020-01-02' """
+            qt_q10 """ select * from user_case_date_with_days_date_partition where d < '2020-01-03' """
+            qt_q11 """ select * from user_case_date_with_days_date_partition where d > '2020-01-01' """
+
+        }
+
         sql """ use `iceberg_catalog`; """
         q01_parquet()
         q01_orc()
+        q01_date()
     }
 }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 05/24: [regression test](schema change) add case for modify partition info (#30338)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 123a187c2397cab78c9144c8ad1248ed021583f2
Author: Guangdong Liu <li...@gmail.com>
AuthorDate: Mon Jan 29 21:55:26 2024 +0800

    [regression test](schema change) add case for modify partition info (#30338)
---
 .../suites/schema_change_p0/test_partition.groovy  | 29 ++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/regression-test/suites/schema_change_p0/test_partition.groovy b/regression-test/suites/schema_change_p0/test_partition.groovy
index d3d00d9feed..ef9e3655302 100644
--- a/regression-test/suites/schema_change_p0/test_partition.groovy
+++ b/regression-test/suites/schema_change_p0/test_partition.groovy
@@ -64,5 +64,34 @@ suite("test_partition") {
     sql """ALTER TABLE ${tblName} DROP PARTITION new_p1;"""
     order_qt_sql_2 """SELECT * from ${tblName};"""
     checkPartition(tblName);
+
+    //modify partition storage_medium
+    sql """ALTER TABLE ${tblName} modify PARTITION old_p1 set("storage_medium"="HDD");"""
+
+    test {
+        sql """show partitions from ${tblName} where partitionName="old_p1";"""
+        check { result, ex, startTime, endTime ->
+            assertEquals("HDD", result[0][10])
+        }
+    }
+
+    //modify partition storage_cooldown_time
+    sql """ALTER TABLE ${tblName} modify PARTITION old_p1 set("storage_medium"="SSD","storage_cooldown_time"="9999-12-31 23:59:59");"""
+
+    test {
+        sql """show partitions from ${tblName} where partitionName="old_p1";"""
+        check { result, ex, startTime, endTime ->
+            assertEquals("SSD", result[0][10])
+            assertEquals("9999-12-31 23:59:59", result[0][11])
+        }
+    }
+
+    //modify partition in_memory
+    test {
+        sql """ALTER TABLE ${tblName} modify PARTITION old_p1 set("in_memory"="true");"""
+        exception "Not support set 'in_memory'='true' now!"
+    }
+
     sql """DROP TABLE IF EXISTS ${tblName} FORCE; """
+
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 19/24: [fix](group commit) Fix group commit VOlapTablePartitionParam memory … (#30491)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 218fb80938e605929060b2d0337ec9b03c90e980
Author: meiyi <my...@gmail.com>
AuthorDate: Tue Jan 30 14:03:29 2024 +0800

    [fix](group commit) Fix group commit VOlapTablePartitionParam memory … (#30491)
---
 be/src/vec/sink/group_commit_block_sink.cpp | 2 +-
 be/src/vec/sink/group_commit_block_sink.h   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/be/src/vec/sink/group_commit_block_sink.cpp b/be/src/vec/sink/group_commit_block_sink.cpp
index 75ffa13233e..cc1abaafcf2 100644
--- a/be/src/vec/sink/group_commit_block_sink.cpp
+++ b/be/src/vec/sink/group_commit_block_sink.cpp
@@ -59,7 +59,7 @@ Status GroupCommitBlockSink::init(const TDataSink& t_sink) {
     _group_commit_mode = table_sink.group_commit_mode;
     _load_id = table_sink.load_id;
     _max_filter_ratio = table_sink.max_filter_ratio;
-    _vpartition = new doris::VOlapTablePartitionParam(_schema, table_sink.partition);
+    _vpartition = std::make_unique<doris::VOlapTablePartitionParam>(_schema, table_sink.partition);
     RETURN_IF_ERROR(_vpartition->init());
     return Status::OK();
 }
diff --git a/be/src/vec/sink/group_commit_block_sink.h b/be/src/vec/sink/group_commit_block_sink.h
index 9a57cb594e7..cfe7727e3e6 100644
--- a/be/src/vec/sink/group_commit_block_sink.h
+++ b/be/src/vec/sink/group_commit_block_sink.h
@@ -73,7 +73,7 @@ private:
     double _max_filter_ratio = 0.0;
 
     // used for find_partition
-    VOlapTablePartitionParam* _vpartition = nullptr;
+    std::unique_ptr<VOlapTablePartitionParam> _vpartition = nullptr;
     // reuse for find_tablet.
     std::vector<VOlapTablePartition*> _partitions;
     Bitmap _filter_bitmap;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 18/24: [fix](Nereids) create constraint write edit log in lock scope (#30422)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit c810df55c509693cd1139b60a986e09a892e4f97
Author: 谢健 <ji...@gmail.com>
AuthorDate: Tue Jan 30 12:04:17 2024 +0800

    [fix](Nereids) create constraint write edit log in lock scope (#30422)
    
    write edit log in lock scope to ensure the order of log sequence.
    
    To avoid the sequence like:
    ```
    add primary key pk1
    add foreign key ref pk1
    log foreign key
    log primary key
    ```
---
 .../java/org/apache/doris/catalog/TableIf.java     | 76 ++++++++++++----------
 .../trees/plans/commands/AddConstraintCommand.java | 13 ++--
 .../plans/commands/DropConstraintCommand.java      |  5 +-
 .../catalog/constraint/ConstraintPersistTest.java  |  7 +-
 4 files changed, 52 insertions(+), 49 deletions(-)

diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index 67efd98fec6..c57d6cce7fb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -26,6 +26,7 @@ import org.apache.doris.cluster.ClusterNamespace;
 import org.apache.doris.common.DdlException;
 import org.apache.doris.common.MetaNotFoundException;
 import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.persist.AlterConstraintLog;
 import org.apache.doris.statistics.AnalysisInfo;
 import org.apache.doris.statistics.BaseAnalysisTask;
 import org.apache.doris.statistics.ColumnStatistic;
@@ -225,7 +226,7 @@ public interface TableIf {
     }
 
     // Note this function is not thread safe
-    default void checkConstraintNotExistence(String name, Constraint primaryKeyConstraint,
+    default void checkConstraintNotExistenceUnsafe(String name, Constraint primaryKeyConstraint,
             Map<String, Constraint> constraintMap) {
         if (constraintMap.containsKey(name)) {
             throw new RuntimeException(String.format("Constraint name %s has existed", name));
@@ -238,63 +239,72 @@ public interface TableIf {
         }
     }
 
-    default Constraint addUniqueConstraint(String name, ImmutableList<String> columns) {
+    default void addUniqueConstraint(String name, ImmutableList<String> columns, boolean replay) {
         writeLock();
         try {
             Map<String, Constraint> constraintMap = getConstraintsMapUnsafe();
             UniqueConstraint uniqueConstraint =  new UniqueConstraint(name, ImmutableSet.copyOf(columns));
-            checkConstraintNotExistence(name, uniqueConstraint, constraintMap);
+            checkConstraintNotExistenceUnsafe(name, uniqueConstraint, constraintMap);
             constraintMap.put(name, uniqueConstraint);
-            return uniqueConstraint;
+            if (!replay) {
+                Env.getCurrentEnv().getEditLog().logAddConstraint(
+                        new AlterConstraintLog(uniqueConstraint, this));
+            }
         } finally {
             writeUnlock();
         }
     }
 
-    default Constraint addPrimaryKeyConstraint(String name, ImmutableList<String> columns) {
+    default void addPrimaryKeyConstraint(String name, ImmutableList<String> columns, boolean replay) {
         writeLock();
         try {
             Map<String, Constraint> constraintMap = getConstraintsMapUnsafe();
             PrimaryKeyConstraint primaryKeyConstraint = new PrimaryKeyConstraint(name, ImmutableSet.copyOf(columns));
-            checkConstraintNotExistence(name, primaryKeyConstraint, constraintMap);
+            checkConstraintNotExistenceUnsafe(name, primaryKeyConstraint, constraintMap);
             constraintMap.put(name, primaryKeyConstraint);
-            return primaryKeyConstraint;
+            if (!replay) {
+                Env.getCurrentEnv().getEditLog().logAddConstraint(
+                        new AlterConstraintLog(primaryKeyConstraint, this));
+            }
         } finally {
             writeUnlock();
         }
     }
 
-    default void updatePrimaryKeyForForeignKey(PrimaryKeyConstraint requirePrimaryKey, TableIf referencedTable) {
-        referencedTable.writeLock();
-        try {
-            Optional<Constraint> primaryKeyConstraint = referencedTable.getConstraintsMapUnsafe().values().stream()
-                    .filter(requirePrimaryKey::equals)
-                    .findFirst();
-            if (!primaryKeyConstraint.isPresent()) {
-                throw new AnalysisException(String.format(
-                        "Foreign key constraint requires a primary key constraint %s in %s",
-                        requirePrimaryKey.getPrimaryKeyNames(), referencedTable.getName()));
-            }
-            ((PrimaryKeyConstraint) (primaryKeyConstraint.get())).addForeignTable(this);
-        } finally {
-            referencedTable.writeUnlock();
+    default PrimaryKeyConstraint tryGetPrimaryKeyForForeignKeyUnsafe(
+            PrimaryKeyConstraint requirePrimaryKey, TableIf referencedTable) {
+        Optional<Constraint> primaryKeyConstraint = referencedTable.getConstraintsMapUnsafe().values().stream()
+                .filter(requirePrimaryKey::equals)
+                .findFirst();
+        if (!primaryKeyConstraint.isPresent()) {
+            throw new AnalysisException(String.format(
+                    "Foreign key constraint requires a primary key constraint %s in %s",
+                    requirePrimaryKey.getPrimaryKeyNames(), referencedTable.getName()));
         }
+        return ((PrimaryKeyConstraint) (primaryKeyConstraint.get()));
     }
 
-    default Constraint addForeignConstraint(String name, ImmutableList<String> columns,
-            TableIf referencedTable, ImmutableList<String> referencedColumns) {
+    default void addForeignConstraint(String name, ImmutableList<String> columns,
+            TableIf referencedTable, ImmutableList<String> referencedColumns, boolean replay) {
         writeLock();
+        referencedTable.writeLock();
         try {
             Map<String, Constraint> constraintMap = getConstraintsMapUnsafe();
             ForeignKeyConstraint foreignKeyConstraint =
                     new ForeignKeyConstraint(name, columns, referencedTable, referencedColumns);
-            checkConstraintNotExistence(name, foreignKeyConstraint, constraintMap);
-            PrimaryKeyConstraint requirePrimaryKey = new PrimaryKeyConstraint(name,
+            checkConstraintNotExistenceUnsafe(name, foreignKeyConstraint, constraintMap);
+            PrimaryKeyConstraint requirePrimaryKeyName = new PrimaryKeyConstraint(name,
                     foreignKeyConstraint.getReferencedColumnNames());
-            updatePrimaryKeyForForeignKey(requirePrimaryKey, referencedTable);
+            PrimaryKeyConstraint primaryKeyConstraint =
+                    tryGetPrimaryKeyForForeignKeyUnsafe(requirePrimaryKeyName, referencedTable);
+            primaryKeyConstraint.addForeignTable(this);
             constraintMap.put(name, foreignKeyConstraint);
-            return foreignKeyConstraint;
+            if (!replay) {
+                Env.getCurrentEnv().getEditLog().logAddConstraint(
+                        new AlterConstraintLog(foreignKeyConstraint, this));
+            }
         } finally {
+            referencedTable.writeUnlock();
             writeUnlock();
         }
     }
@@ -303,22 +313,21 @@ public interface TableIf {
         if (constraint instanceof UniqueConstraint) {
             UniqueConstraint uniqueConstraint = (UniqueConstraint) constraint;
             this.addUniqueConstraint(constraint.getName(),
-                    ImmutableList.copyOf(uniqueConstraint.getUniqueColumnNames()));
+                    ImmutableList.copyOf(uniqueConstraint.getUniqueColumnNames()), true);
         } else if (constraint instanceof PrimaryKeyConstraint) {
             PrimaryKeyConstraint primaryKeyConstraint = (PrimaryKeyConstraint) constraint;
             this.addPrimaryKeyConstraint(primaryKeyConstraint.getName(),
-                    ImmutableList.copyOf(primaryKeyConstraint.getPrimaryKeyNames()));
+                    ImmutableList.copyOf(primaryKeyConstraint.getPrimaryKeyNames()), true);
         } else if (constraint instanceof ForeignKeyConstraint) {
             ForeignKeyConstraint foreignKey = (ForeignKeyConstraint) constraint;
             this.addForeignConstraint(foreignKey.getName(),
                     ImmutableList.copyOf(foreignKey.getForeignKeyNames()),
                     foreignKey.getReferencedTable(),
-                    ImmutableList.copyOf(foreignKey.getReferencedColumnNames()));
+                    ImmutableList.copyOf(foreignKey.getReferencedColumnNames()), true);
         }
     }
 
-    default Constraint dropConstraint(String name) {
-        Constraint dropConstraint;
+    default void dropConstraint(String name) {
         writeLock();
         try {
             Map<String, Constraint> constraintMap = getConstraintsMapUnsafe();
@@ -332,11 +341,10 @@ public interface TableIf {
                 ((PrimaryKeyConstraint) constraint).getForeignTables()
                         .forEach(t -> t.dropFKReferringPK(this, (PrimaryKeyConstraint) constraint));
             }
-            dropConstraint = constraint;
+            Env.getCurrentEnv().getEditLog().logDropConstraint(new AlterConstraintLog(constraint, this));
         } finally {
             writeUnlock();
         }
-        return dropConstraint;
     }
 
     default void dropFKReferringPK(TableIf table, PrimaryKeyConstraint constraint) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AddConstraintCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AddConstraintCommand.java
index 8c90bc0f914..38bf71133c0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AddConstraintCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AddConstraintCommand.java
@@ -17,7 +17,6 @@
 
 package org.apache.doris.nereids.trees.plans.commands;
 
-import org.apache.doris.catalog.Env;
 import org.apache.doris.catalog.TableIf;
 import org.apache.doris.common.Pair;
 import org.apache.doris.nereids.NereidsPlanner;
@@ -30,7 +29,6 @@ import org.apache.doris.nereids.trees.plans.commands.ExplainCommand.ExplainLevel
 import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
 import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
 import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
-import org.apache.doris.persist.AlterConstraintLog;
 import org.apache.doris.qe.ConnectContext;
 import org.apache.doris.qe.StmtExecutor;
 
@@ -63,19 +61,16 @@ public class AddConstraintCommand extends Command implements ForwardWithSync {
     @Override
     public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
         Pair<ImmutableList<String>, TableIf> columnsAndTable = extractColumnsAndTable(ctx, constraint.toProject());
-        org.apache.doris.catalog.constraint.Constraint catalogConstraint = null;
         if (constraint.isForeignKey()) {
             Pair<ImmutableList<String>, TableIf> referencedColumnsAndTable
                     = extractColumnsAndTable(ctx, constraint.toReferenceProject());
-            catalogConstraint = columnsAndTable.second.addForeignConstraint(name, columnsAndTable.first,
-                    referencedColumnsAndTable.second, referencedColumnsAndTable.first);
+            columnsAndTable.second.addForeignConstraint(name, columnsAndTable.first,
+                    referencedColumnsAndTable.second, referencedColumnsAndTable.first, false);
         } else if (constraint.isPrimaryKey()) {
-            catalogConstraint = columnsAndTable.second.addPrimaryKeyConstraint(name, columnsAndTable.first);
+            columnsAndTable.second.addPrimaryKeyConstraint(name, columnsAndTable.first, false);
         } else if (constraint.isUnique()) {
-            catalogConstraint = columnsAndTable.second.addUniqueConstraint(name, columnsAndTable.first);
+            columnsAndTable.second.addUniqueConstraint(name, columnsAndTable.first, false);
         }
-        Env.getCurrentEnv().getEditLog().logAddConstraint(
-                new AlterConstraintLog(catalogConstraint, columnsAndTable.second));
     }
 
     private Pair<ImmutableList<String>, TableIf> extractColumnsAndTable(ConnectContext ctx, LogicalPlan plan) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropConstraintCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropConstraintCommand.java
index 84143f234c3..d13ca98ef90 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropConstraintCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DropConstraintCommand.java
@@ -17,7 +17,6 @@
 
 package org.apache.doris.nereids.trees.plans.commands;
 
-import org.apache.doris.catalog.Env;
 import org.apache.doris.catalog.TableIf;
 import org.apache.doris.nereids.NereidsPlanner;
 import org.apache.doris.nereids.exceptions.AnalysisException;
@@ -28,7 +27,6 @@ import org.apache.doris.nereids.trees.plans.commands.ExplainCommand.ExplainLevel
 import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
 import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
 import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
-import org.apache.doris.persist.AlterConstraintLog;
 import org.apache.doris.qe.ConnectContext;
 import org.apache.doris.qe.StmtExecutor;
 
@@ -58,8 +56,7 @@ public class DropConstraintCommand extends Command implements ForwardWithSync {
     @Override
     public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
         TableIf table = extractTable(ctx, plan);
-        org.apache.doris.catalog.constraint.Constraint catalogConstraint = table.dropConstraint(name);
-        Env.getCurrentEnv().getEditLog().logDropConstraint(new AlterConstraintLog(catalogConstraint, table));
+        table.dropConstraint(name);
     }
 
     private TableIf extractTable(ConnectContext ctx, LogicalPlan plan) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/constraint/ConstraintPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/constraint/ConstraintPersistTest.java
index 64f3db583ad..045d1893022 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/constraint/ConstraintPersistTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/constraint/ConstraintPersistTest.java
@@ -169,8 +169,11 @@ class ConstraintPersistTest extends TestWithFeService implements PlanPatternMatc
     @Test
     void externalTableTest() throws Exception {
         ExternalTable externalTable =  new ExternalTable();
-        externalTable.addPrimaryKeyConstraint("pk", ImmutableList.of("col"));
-
+        try {
+            externalTable.addPrimaryKeyConstraint("pk", ImmutableList.of("col"), false);
+        } catch (Exception ignore) {
+            // ignore
+        }
         ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
         DataOutput output = new DataOutputStream(outputStream);
         externalTable.write(output);


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 03/24: [bug](node) add dependency for set operation node (#30203)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit f7e01ceffa863dcf548df43afb94f1bb6f9d5167
Author: zhangstar333 <87...@users.noreply.github.com>
AuthorDate: Mon Jan 29 21:21:02 2024 +0800

    [bug](node) add dependency for set operation node (#30203)
    
    These sinks must be completed one by one in order, eg: child(1) must wait child(0) build finish
---
 be/src/pipeline/pipeline_fragment_context.cpp      |  6 +-
 be/src/vec/exec/vset_operation_node.cpp            | 11 ++--
 .../data/nereids_syntax_p0/set_operation.out       |  4 ++
 .../suites/nereids_syntax_p0/set_operation.groovy  | 74 ++++++++++++++++++++++
 4 files changed, 90 insertions(+), 5 deletions(-)

diff --git a/be/src/pipeline/pipeline_fragment_context.cpp b/be/src/pipeline/pipeline_fragment_context.cpp
index 5e8af4940b8..64572f03692 100644
--- a/be/src/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_fragment_context.cpp
@@ -693,7 +693,8 @@ Status PipelineFragmentContext::_build_operators_for_set_operation_node(ExecNode
     OperatorBuilderPtr sink_builder =
             std::make_shared<SetSinkOperatorBuilder<is_intersect>>(node->id(), node);
     RETURN_IF_ERROR(build_pipeline->set_sink_builder(sink_builder));
-
+    std::vector<PipelinePtr> all_pipelines;
+    all_pipelines.emplace_back(build_pipeline);
     for (int child_id = 1; child_id < node->children_count(); ++child_id) {
         auto probe_pipeline = add_pipeline();
         RETURN_IF_ERROR(_build_pipelines(node->child(child_id), probe_pipeline));
@@ -701,6 +702,9 @@ Status PipelineFragmentContext::_build_operators_for_set_operation_node(ExecNode
                 std::make_shared<SetProbeSinkOperatorBuilder<is_intersect>>(node->id(), child_id,
                                                                             node);
         RETURN_IF_ERROR(probe_pipeline->set_sink_builder(probe_sink_builder));
+        //eg: These sinks must be completed one by one in order, child(1) must wait child(0) build finish
+        probe_pipeline->add_dependency(all_pipelines[child_id - 1]);
+        all_pipelines.emplace_back(probe_pipeline);
     }
 
     OperatorBuilderPtr source_builder =
diff --git a/be/src/vec/exec/vset_operation_node.cpp b/be/src/vec/exec/vset_operation_node.cpp
index 75317b4c933..16c267b26df 100644
--- a/be/src/vec/exec/vset_operation_node.cpp
+++ b/be/src/vec/exec/vset_operation_node.cpp
@@ -352,10 +352,13 @@ Status VSetOperationNode<is_intersect>::sink_probe(RuntimeState* state, int chil
                                                    bool eos) {
     SCOPED_TIMER(_exec_timer);
     SCOPED_TIMER(_probe_timer);
-    CHECK(_build_finished) << "cannot sink probe data before build finished";
-    if (child_id > 1) {
-        CHECK(_probe_finished_children_index[child_id - 1])
-                << fmt::format("child with id: {} should be probed first", child_id);
+    if (!_build_finished) {
+        return Status::RuntimeError("cannot sink probe data before build finished " +
+                                    std::to_string(child_id));
+    }
+    if (child_id > 1 && !_probe_finished_children_index[child_id - 1]) {
+        return Status::RuntimeError("the child with id should be probed first " +
+                                    std::to_string(child_id - 1));
     }
     auto probe_rows = block->rows();
     if (probe_rows > 0) {
diff --git a/regression-test/data/nereids_syntax_p0/set_operation.out b/regression-test/data/nereids_syntax_p0/set_operation.out
index 28f286ea3d6..0ce9a4b4c99 100644
--- a/regression-test/data/nereids_syntax_p0/set_operation.out
+++ b/regression-test/data/nereids_syntax_p0/set_operation.out
@@ -596,3 +596,7 @@ hell0
 2.0554876421875E8	3601
 5.395085565625E7	3602
 
+-- !intersect_case --
+0
+1
+
diff --git a/regression-test/suites/nereids_syntax_p0/set_operation.groovy b/regression-test/suites/nereids_syntax_p0/set_operation.groovy
index b23811727bf..6d7f4e18ff8 100644
--- a/regression-test/suites/nereids_syntax_p0/set_operation.groovy
+++ b/regression-test/suites/nereids_syntax_p0/set_operation.groovy
@@ -329,4 +329,78 @@ suite("test_nereids_set_operation") {
         union
         select avg(tap), potno from dwd_daytable where potno=3602 and ddate >= '2023-08-01' group by potno limit 10;
     """
+
+    sql "DROP TABLE IF EXISTS table_22_undef_partitions2_keys3_properties4_distributed_by54"
+    sql """
+    create table table_22_undef_partitions2_keys3_properties4_distributed_by54 (
+        `col_int_undef_signed_not_null` int  not null ,
+        `col_date_undef_signed_not_null` date  not null ,
+        `pk` int,
+        `col_int_undef_signed` int  null ,
+        `col_date_undef_signed` date  null ,
+        `col_varchar_10__undef_signed` varchar(10)  null ,
+        `col_varchar_10__undef_signed_not_null` varchar(10)  not null ,
+        `col_varchar_1024__undef_signed` varchar(1024)  null ,
+        `col_varchar_1024__undef_signed_not_null` varchar(1024)  not null 
+    ) engine=olap
+    DUPLICATE KEY(col_int_undef_signed_not_null, col_date_undef_signed_not_null, pk)
+    PARTITION BY RANGE(col_int_undef_signed_not_null, col_date_undef_signed_not_null) (PARTITION p0 VALUES [('-10000', '2023-12-01'), ('3', '2023-12-10')), PARTITION p1 VALUES [('3', '2023-12-10'), ('6', '2023-12-15')), PARTITION p2 VALUES [('6', '2023-12-15'), ('10000', '2023-12-21')))
+    distributed by hash(pk) buckets 10
+    properties("replication_num" = "1");
+    """
+
+    sql """
+        insert into table_22_undef_partitions2_keys3_properties4_distributed_by54(pk,col_int_undef_signed,col_int_undef_signed_not_null,col_date_undef_signed,col_date_undef_signed_not_null,col_varchar_10__undef_signed,col_varchar_10__undef_signed_not_null,col_varchar_1024__undef_signed,col_varchar_1024__undef_signed_not_null) values (0,0,2,"2023-12-10","2023-12-09","ok",'f','a','l'),(1,1,9,"2023-12-19","2023-12-10",'x',"it","is",'b'),(2,null,7,"2023-12-18","2023-12-13","on",'p',"why",'x' [...]
+    """
+
+    sql "DROP TABLE IF EXISTS table_3_undef_partitions2_keys3_properties4_distributed_by54"
+    sql """
+    create table table_3_undef_partitions2_keys3_properties4_distributed_by54 (
+        `col_int_undef_signed_not_null` int  not null ,
+        `col_date_undef_signed_not_null` date  not null ,
+        `pk` int,
+        `col_int_undef_signed` int MIN  null ,
+        `col_date_undef_signed` date REPLACE  null ,
+        `col_varchar_10__undef_signed` varchar(10) REPLACE  null ,
+        `col_varchar_10__undef_signed_not_null` varchar(10) MIN  not null ,
+        `col_varchar_1024__undef_signed` varchar(1024) REPLACE  null ,
+        `col_varchar_1024__undef_signed_not_null` varchar(1024) REPLACE  not null 
+    ) engine=olap
+    AGGREGATE KEY(col_int_undef_signed_not_null, col_date_undef_signed_not_null, pk)
+    PARTITION BY RANGE(col_int_undef_signed_not_null, col_date_undef_signed_not_null) (PARTITION p0 VALUES [('-10000', '2023-12-01'), ('3', '2023-12-10')), PARTITION p1 VALUES [('3', '2023-12-10'), ('6', '2023-12-15')), PARTITION p2 VALUES [('6', '2023-12-15'), ('10000', '2023-12-21')))
+    distributed by hash(pk) buckets 10
+    properties("replication_num" = "1");
+    """
+
+    sql """
+        insert into table_3_undef_partitions2_keys3_properties4_distributed_by54(pk,col_int_undef_signed,col_int_undef_signed_not_null,col_date_undef_signed,col_date_undef_signed_not_null,col_varchar_10__undef_signed,col_varchar_10__undef_signed_not_null,col_varchar_1024__undef_signed,col_varchar_1024__undef_signed_not_null) values (0,null,5,"2023-12-13","2023-12-15",'c','c',"ok","had"),(1,null,7,"2023-12-12","2023-12-19","up",'e',"why",'c'),(2,3,2,"2023-12-15","2023-12-18","hey",'b',"as [...]
+    """
+
+
+    sql "DROP TABLE IF EXISTS table_2_undef_partitions2_keys3_properties4_distributed_by55"
+    sql """
+    create table table_2_undef_partitions2_keys3_properties4_distributed_by55 (
+        `col_int_undef_signed_not_null` int  not null ,
+        `col_date_undef_signed_not_null` date  not null ,
+        `pk` int,
+        `col_int_undef_signed` int REPLACE  null ,
+        `col_date_undef_signed` date MIN  null ,
+        `col_varchar_10__undef_signed` varchar(10) MIN  null ,
+        `col_varchar_10__undef_signed_not_null` varchar(10) MAX  not null ,
+        `col_varchar_1024__undef_signed` varchar(1024) REPLACE  null ,
+        `col_varchar_1024__undef_signed_not_null` varchar(1024) MIN  not null 
+    ) engine=olap
+    AGGREGATE KEY(col_int_undef_signed_not_null, col_date_undef_signed_not_null, pk)
+    PARTITION BY RANGE(col_int_undef_signed_not_null, col_date_undef_signed_not_null) (PARTITION p0 VALUES [('-10000', '2023-12-01'), ('3', '2023-12-10')), PARTITION p1 VALUES [('3', '2023-12-10'), ('6', '2023-12-15')), PARTITION p2 VALUES [('6', '2023-12-15'), ('10000', '2023-12-21')))
+    distributed by hash(pk) buckets 10
+    properties("replication_num" = "1");
+    """
+
+    sql """
+        insert into table_2_undef_partitions2_keys3_properties4_distributed_by55(pk,col_int_undef_signed,col_int_undef_signed_not_null,col_date_undef_signed,col_date_undef_signed_not_null,col_varchar_10__undef_signed,col_varchar_10__undef_signed_not_null,col_varchar_1024__undef_signed,col_varchar_1024__undef_signed_not_null) values (0,null,2,"2023-12-20","2023-12-11",'m','g','t',"in"),(1,1,4,"2023-12-09","2023-12-19","had",'b',"was","didn't");
+    """
+
+    qt_intersect_case """
+        SELECT subq1.`pk` AS pk1 FROM ( (  SELECT t1.`pk`  FROM table_22_undef_partitions2_keys3_properties4_distributed_by54 AS t1 INNER JOIN table_3_undef_partitions2_keys3_properties4_distributed_by54 AS alias1 ON t1 . `pk` = alias1 . `pk`   ) INTERSECT (  SELECT t1.`pk`  FROM table_22_undef_partitions2_keys3_properties4_distributed_by54 AS t1 INNER JOIN table_2_undef_partitions2_keys3_properties4_distributed_by55 AS alias2 ON t1 . `pk` = alias2 . `pk`   ) ) subq1 GROUP BY subq1.`pk`  [...]
+    """
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


(doris) 24/24: [fix](regex) use boost regex instead of std (#30462)

Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 7fe4d00bb21826c69d76f927dfca3c157f2e7ba9
Author: Yongqiang YANG <98...@users.noreply.github.com>
AuthorDate: Tue Jan 30 15:26:30 2024 +0800

    [fix](regex) use boost regex instead of std (#30462)
---
 be/src/olap/delete_handler.cpp | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/be/src/olap/delete_handler.cpp b/be/src/olap/delete_handler.cpp
index f8b22b6a0d7..29f84563627 100644
--- a/be/src/olap/delete_handler.cpp
+++ b/be/src/olap/delete_handler.cpp
@@ -22,8 +22,8 @@
 #include <thrift/protocol/TDebugProtocol.h>
 
 #include <algorithm>
+#include <boost/regex.hpp>
 #include <limits>
-#include <regex>
 #include <sstream>
 #include <string>
 #include <utility>
@@ -44,10 +44,10 @@ using std::vector;
 using std::string;
 using std::stringstream;
 
-using std::regex;
-using std::regex_error;
-using std::regex_match;
-using std::smatch;
+using boost::regex;
+using boost::regex_error;
+using boost::regex_match;
+using boost::smatch;
 
 using ::google::protobuf::RepeatedPtrField;
 
@@ -293,7 +293,7 @@ Status DeleteHandler::parse_condition(const DeleteSubPredicatePB& sub_cond, TCon
 
 Status DeleteHandler::parse_condition(const std::string& condition_str, TCondition* condition) {
     bool matched = true;
-    smatch what;
+    boost::smatch what;
 
     try {
         // Condition string format, the format is (column_name)(op)(value)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org