You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/02/26 11:32:57 UTC
(doris) branch branch-2.1 updated (e41d091a714 -> 743eb45695d)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
discard e41d091a714 [Improvement](executor)add remote scan thread pool (#31376)
discard fbbaf74b778 [Improment](executor)Add scanbytes/scanrows condition (#31364)
discard 7b9e500c31a Fix cgroup cpu controller file leak (#31398)
discard 90f36ee1815 [feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823)
discard e0cc087f379 [feature](doris compose) Add create cloud cluster (#31315)
discard c9f8111ec75 [fix](function) fix unexpected be core in string search function (#31312)
discard 17f042b85af [opt](scanner) scan enough blocks in each scan task (#31277)
new b8dafb20017 [fix](function) fix unexpected be core in string search function (#31312)
new 3ba4fcdccc2 [feature](doris compose) Add create cloud cluster (#31315)
new 344a7688ed4 [feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823)
new 602f05b5215 Fix cgroup cpu controller file leak (#31398)
new 2a10c5d2f65 [Improment](executor)Add scanbytes/scanrows condition (#31364)
new 743eb45695d [Improvement](executor)add remote scan thread pool (#31376)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (e41d091a714)
\
N -- N -- N refs/heads/branch-2.1 (743eb45695d)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/vec/exec/scan/scanner_context.cpp | 151 +++++++++++++++++------------
be/src/vec/exec/scan/scanner_context.h | 27 ++++--
be/src/vec/exec/scan/scanner_scheduler.cpp | 36 +++----
3 files changed, 124 insertions(+), 90 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/06: [feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 344a7688ed4e386f35df28dd4c7580ec8ac59286
Author: Chester <42...@users.noreply.github.com>
AuthorDate: Mon Feb 26 17:33:20 2024 +0800
[feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823)
---
.../vec/functions/array/function_array_range.cpp | 184 +++++++++++++++----
.../sql-functions/array-functions/array-range.md | 27 ++-
.../sql-functions/array-functions/sequence.md | 104 +++++++++++
docs/sidebars.json | 3 +-
.../sql-functions/array-functions/array-range.md | 27 ++-
.../sql-functions/array-functions/sequence.md | 102 +++++++++++
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 2 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 9 +
.../apache/doris/analysis/FunctionCallExpr.java | 7 +
.../doris/catalog/BuiltinScalarFunctions.java | 2 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 43 +++++
.../expressions/functions/scalar/ArrayRange.java | 7 +-
.../{ArrayRange.java => ArrayRangeDayUnit.java} | 57 ++----
.../{ArrayRange.java => ArrayRangeHourUnit.java} | 57 ++----
.../{ArrayRange.java => ArrayRangeMinuteUnit.java} | 57 ++----
.../{ArrayRange.java => ArrayRangeMonthUnit.java} | 57 ++----
.../{ArrayRange.java => ArrayRangeSecondUnit.java} | 57 ++----
.../{ArrayRange.java => ArrayRangeWeekUnit.java} | 57 ++----
.../{ArrayRange.java => ArrayRangeYearUnit.java} | 57 ++----
.../expressions/visitor/ScalarFunctionVisitor.java | 35 ++++
gensrc/script/doris_builtins_functions.py | 8 +-
.../nereids_function_p0/scalar_function/Array.out | 194 +++++++++++++++++++++
.../array_functions/test_array_functions.out | 131 ++++++++++++++
.../scalar_function/Array.groovy | 14 ++
.../array_functions/test_array_functions.groovy | 84 +++++++++
25 files changed, 1056 insertions(+), 326 deletions(-)
diff --git a/be/src/vec/functions/array/function_array_range.cpp b/be/src/vec/functions/array/function_array_range.cpp
index 2431e76c889..a84b94d8cfc 100644
--- a/be/src/vec/functions/array/function_array_range.cpp
+++ b/be/src/vec/functions/array/function_array_range.cpp
@@ -38,10 +38,13 @@
#include "vec/core/types.h"
#include "vec/data_types/data_type.h"
#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_date_time.h"
#include "vec/data_types/data_type_nullable.h"
#include "vec/data_types/data_type_number.h"
#include "vec/functions/function.h"
+#include "vec/functions/function_date_or_datetime_computation.h"
#include "vec/functions/simple_function_factory.h"
+#include "vec/runtime/vdatetime_value.h"
#include "vec/utils/util.hpp"
namespace doris {
@@ -53,7 +56,7 @@ namespace doris::vectorized {
template <typename Impl>
class FunctionArrayRange : public IFunction {
public:
- static constexpr auto name = "array_range";
+ static constexpr auto name = Impl::name;
static FunctionPtr create() { return std::make_shared<FunctionArrayRange>(); }
@@ -73,7 +76,7 @@ public:
}
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
- auto nested_type = make_nullable(std::make_shared<DataTypeInt32>());
+ auto nested_type = make_nullable(Impl::get_data_type());
auto res = std::make_shared<DataTypeArray>(nested_type);
return make_nullable(res);
}
@@ -84,11 +87,52 @@ public:
}
};
+template <typename SourceDataType, typename TimeUnitOrVoid = void>
struct RangeImplUtil {
+ using DataType = std::conditional_t<std::is_same_v<SourceDataType, Int32>, DataTypeInt32,
+ DataTypeDateTimeV2>;
+
+ static DataTypePtr get_data_type() { return std::make_shared<DataType>(); }
+
+ static constexpr const char* get_function_name() {
+ if constexpr (std::is_same_v<SourceDataType, DateTimeV2> &&
+ !std::is_same_v<TimeUnitOrVoid, void>) {
+ if constexpr (std::is_same_v<TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::YEAR>>) {
+ return "array_range_year_unit";
+ } else if constexpr (std::is_same_v<
+ TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::MONTH>>) {
+ return "array_range_month_unit";
+ } else if constexpr (std::is_same_v<TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::WEEK>>) {
+ return "array_range_week_unit";
+ } else if constexpr (std::is_same_v<TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::DAY>>) {
+ return "array_range_day_unit";
+ } else if constexpr (std::is_same_v<TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::HOUR>>) {
+ return "array_range_hour_unit";
+ } else if constexpr (std::is_same_v<
+ TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::MINUTE>>) {
+ return "array_range_minute_unit";
+ } else if constexpr (std::is_same_v<
+ TimeUnitOrVoid,
+ std::integral_constant<TimeUnit, TimeUnit::SECOND>>) {
+ return "array_range_second_unit";
+ }
+ } else {
+ return "array_range";
+ }
+ }
+
+ static constexpr auto name = get_function_name();
+
static Status range_execute(Block& block, const ColumnNumbers& arguments, size_t result,
size_t input_rows_count) {
DCHECK_EQ(arguments.size(), 3);
- auto return_nested_type = make_nullable(std::make_shared<DataTypeInt32>());
+ auto return_nested_type = make_nullable(std::make_shared<DataType>());
auto dest_array_column_ptr = ColumnArray::create(return_nested_type->create_column(),
ColumnArray::ColumnOffsets::create());
IColumn* dest_nested_column = &dest_array_column_ptr->get_data();
@@ -111,13 +155,15 @@ struct RangeImplUtil {
argument_columns[i] = nullable->get_nested_column_ptr();
}
}
- auto start_column = assert_cast<const ColumnVector<Int32>*>(argument_columns[0].get());
- auto end_column = assert_cast<const ColumnVector<Int32>*>(argument_columns[1].get());
+ auto start_column =
+ assert_cast<const ColumnVector<SourceDataType>*>(argument_columns[0].get());
+ auto end_column =
+ assert_cast<const ColumnVector<SourceDataType>*>(argument_columns[1].get());
auto step_column = assert_cast<const ColumnVector<Int32>*>(argument_columns[2].get());
DCHECK(dest_nested_column != nullptr);
auto& dest_offsets = dest_array_column_ptr->get_offsets();
- auto nested_column = reinterpret_cast<ColumnVector<Int32>*>(dest_nested_column);
+ auto nested_column = reinterpret_cast<ColumnVector<SourceDataType>*>(dest_nested_column);
dest_offsets.reserve(input_rows_count);
dest_nested_column->reserve(input_rows_count);
dest_nested_null_map.reserve(input_rows_count);
@@ -132,49 +178,92 @@ struct RangeImplUtil {
}
private:
- static void vector(const PaddedPODArray<Int32>& start, const PaddedPODArray<Int32>& end,
- const PaddedPODArray<Int32>& step, NullMap& args_null_map,
- PaddedPODArray<Int32>& nested_column,
+ static void vector(const PaddedPODArray<SourceDataType>& start,
+ const PaddedPODArray<SourceDataType>& end, const PaddedPODArray<Int32>& step,
+ NullMap& args_null_map, PaddedPODArray<SourceDataType>& nested_column,
PaddedPODArray<UInt8>& dest_nested_null_map,
ColumnArray::Offsets64& dest_offsets) {
int rows = start.size();
for (auto row = 0; row < rows; ++row) {
- if (args_null_map[row] || start[row] < 0 || end[row] < 0 || step[row] < 0) {
- nested_column.push_back(0);
- dest_offsets.push_back(dest_offsets.back() + 1);
- dest_nested_null_map.push_back(1);
- args_null_map[row] = 1;
+ auto idx = start[row];
+ auto end_row = end[row];
+ auto step_row = step[row];
+ auto args_null_map_row = args_null_map[row];
+ if constexpr (std::is_same_v<SourceDataType, Int32>) {
+ if (args_null_map_row || idx < 0 || end_row < 0 || step_row <= 0) {
+ args_null_map[row] = 1;
+ dest_offsets.push_back(dest_offsets.back());
+ continue;
+ } else {
+ int offset = dest_offsets.back();
+ while (idx < end[row]) {
+ nested_column.push_back(idx);
+ dest_nested_null_map.push_back(0);
+ offset++;
+ idx = idx + step_row;
+ }
+ dest_offsets.push_back(offset);
+ }
} else {
- int offset = dest_offsets.back();
- for (auto idx = start[row]; idx < end[row]; idx = idx + step[row]) {
- nested_column.push_back(idx);
- dest_nested_null_map.push_back(0);
- offset++;
+ const auto& idx_0 = reinterpret_cast<const DateV2Value<DateTimeV2ValueType>&>(idx);
+ const auto& end_row_cast =
+ reinterpret_cast<const DateV2Value<DateTimeV2ValueType>&>(end_row);
+ bool is_null = !idx_0.is_valid_date();
+ bool is_end_row_invalid = !end_row_cast.is_valid_date();
+ if (args_null_map_row || step_row <= 0 || is_null || is_end_row_invalid) {
+ args_null_map[row] = 1;
+ dest_offsets.push_back(dest_offsets.back());
+ continue;
+ } else {
+ int offset = dest_offsets.back();
+ using UNIT = std::conditional_t<std::is_same_v<TimeUnitOrVoid, void>,
+ std::integral_constant<TimeUnit, TimeUnit::DAY>,
+ TimeUnitOrVoid>;
+ while (doris::datetime_diff<UNIT::value, DateTimeV2ValueType,
+ DateTimeV2ValueType>(idx, end_row) > 0) {
+ nested_column.push_back(idx);
+ dest_nested_null_map.push_back(0);
+ offset++;
+ idx = doris::vectorized::date_time_add<
+ UNIT::value, DateV2Value<DateTimeV2ValueType>,
+ DateV2Value<DateTimeV2ValueType>, DateTimeV2>(idx, step_row,
+ is_null);
+ }
+ dest_offsets.push_back(offset);
}
- dest_offsets.push_back(offset);
}
}
}
};
-struct RangeOneImpl {
- static DataTypes get_variadic_argument_types() { return {std::make_shared<DataTypeInt32>()}; }
+template <typename SourceDataType, typename TimeUnitOrVoid = void>
+struct RangeOneImpl : public RangeImplUtil<SourceDataType, TimeUnitOrVoid> {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<typename RangeImplUtil<SourceDataType>::DataType>()};
+ }
static Status execute_impl(FunctionContext* context, Block& block,
const ColumnNumbers& arguments, size_t result,
size_t input_rows_count) {
- auto start_column = ColumnInt32::create(input_rows_count, 0);
+ using ColumnType = std::conditional_t<std::is_same_v<SourceDataType, Int32>, ColumnInt32,
+ ColumnDateTimeV2>;
+ auto start_column = ColumnType::create(input_rows_count, 0);
auto step_column = ColumnInt32::create(input_rows_count, 1);
- block.insert({std::move(start_column), std::make_shared<DataTypeInt32>(), "start_column"});
+ block.insert({std::move(start_column),
+ std::make_shared<typename RangeImplUtil<SourceDataType>::DataType>(),
+ "start_column"});
block.insert({std::move(step_column), std::make_shared<DataTypeInt32>(), "step_column"});
ColumnNumbers temp_arguments = {block.columns() - 2, arguments[0], block.columns() - 1};
- return RangeImplUtil::range_execute(block, temp_arguments, result, input_rows_count);
+ return (RangeImplUtil<SourceDataType, TimeUnitOrVoid>::range_execute)(
+ block, temp_arguments, result, input_rows_count);
}
};
-struct RangeTwoImpl {
+template <typename SourceDataType, typename TimeUnitOrVoid = void>
+struct RangeTwoImpl : public RangeImplUtil<SourceDataType, TimeUnitOrVoid> {
static DataTypes get_variadic_argument_types() {
- return {std::make_shared<DataTypeInt32>(), std::make_shared<DataTypeInt32>()};
+ return {std::make_shared<typename RangeImplUtil<SourceDataType>::DataType>(),
+ std::make_shared<typename RangeImplUtil<SourceDataType>::DataType>()};
}
static Status execute_impl(FunctionContext* context, Block& block,
@@ -183,27 +272,54 @@ struct RangeTwoImpl {
auto step_column = ColumnInt32::create(input_rows_count, 1);
block.insert({std::move(step_column), std::make_shared<DataTypeInt32>(), "step_column"});
ColumnNumbers temp_arguments = {arguments[0], arguments[1], block.columns() - 1};
- return RangeImplUtil::range_execute(block, temp_arguments, result, input_rows_count);
+ return (RangeImplUtil<SourceDataType, TimeUnitOrVoid>::range_execute)(
+ block, temp_arguments, result, input_rows_count);
}
};
-struct RangeThreeImpl {
+template <typename SourceDataType, typename TimeUnitOrVoid = void>
+struct RangeThreeImpl : public RangeImplUtil<SourceDataType, TimeUnitOrVoid> {
static DataTypes get_variadic_argument_types() {
- return {std::make_shared<DataTypeInt32>(), std::make_shared<DataTypeInt32>(),
+ return {std::make_shared<typename RangeImplUtil<SourceDataType>::DataType>(),
+ std::make_shared<typename RangeImplUtil<SourceDataType>::DataType>(),
std::make_shared<DataTypeInt32>()};
}
static Status execute_impl(FunctionContext* context, Block& block,
const ColumnNumbers& arguments, size_t result,
size_t input_rows_count) {
- return RangeImplUtil::range_execute(block, arguments, result, input_rows_count);
+ return (RangeImplUtil<SourceDataType, TimeUnitOrVoid>::range_execute)(
+ block, arguments, result, input_rows_count);
}
};
void register_function_array_range(SimpleFunctionFactory& factory) {
- factory.register_function<FunctionArrayRange<RangeOneImpl>>();
- factory.register_function<FunctionArrayRange<RangeTwoImpl>>();
- factory.register_function<FunctionArrayRange<RangeThreeImpl>>();
+ /// One argument, just for Int32
+ factory.register_function<FunctionArrayRange<RangeOneImpl<Int32>>>();
+
+ /// Two arguments, for Int32 and DateTimeV2 without Interval
+ factory.register_function<FunctionArrayRange<RangeTwoImpl<Int32>>>();
+ factory.register_function<FunctionArrayRange<RangeTwoImpl<DateTimeV2>>>();
+
+ /// Three arguments, for Int32 and DateTimeV2 with YEAR to SECOND Interval
+ factory.register_function<FunctionArrayRange<RangeThreeImpl<Int32>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::YEAR>>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::MONTH>>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::WEEK>>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::DAY>>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::HOUR>>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::MINUTE>>>>();
+ factory.register_function<FunctionArrayRange<
+ RangeThreeImpl<DateTimeV2, std::integral_constant<TimeUnit, TimeUnit::SECOND>>>>();
+
+ // alias
+ factory.register_alias("array_range", "sequence");
}
} // namespace doris::vectorized
diff --git a/docs/en/docs/sql-manual/sql-functions/array-functions/array-range.md b/docs/en/docs/sql-manual/sql-functions/array-functions/array-range.md
index 0ccbc63729e..c8119a9defe 100644
--- a/docs/en/docs/sql-manual/sql-functions/array-functions/array-range.md
+++ b/docs/en/docs/sql-manual/sql-functions/array-functions/array-range.md
@@ -40,21 +40,28 @@ array_range
ARRAY<Int> array_range(Int end)
ARRAY<Int> array_range(Int start, Int end)
ARRAY<Int> array_range(Int start, Int end, Int step)
+ARRAY<Datetime> array_range(Datetime start_datetime, Datetime end_datetime)
+ARRAY<Datetime> array_range(Datetime start_datetime, Datetime end_datetime, INTERVAL Int interval_step UNIT)
```
+1. To generate array of int:
The parameters are all positive integers.
start default value is 0, and step default value is 1.
Return the array which numbers from start to end - 1 by step.
+2. To generate array of datetime:
+At least taking two parameters.
+The first two parameters are all datetimev2, the third is positive integer.
+If the third part is missing, `INTERVAL 1 DAY` will be default value.
+UNIT supports YEAR/MONTH/WEEK/DAY/HOUR/MINUTE/SECOND.
+Return the array of datetimev2 between start_datetime and closest to end_datetime by interval_step UNIT.
### notice
-`Only supported in vectorized engine`
+`if the 3rd parameter step/interval_step is negative or zero, the function will return NULL`
### example
```
-mysql> set enable_vectorized_engine=true;
-
mysql> select array_range(10);
+--------------------------------+
| array_range(10) |
@@ -75,6 +82,20 @@ mysql> select array_range(0,20,2);
+-------------------------------------+
| [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] |
+-------------------------------------+
+
+mysql> select array_range(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0))) AS array_range_default;
++------------------------------------------------+
+| array_range_default |
++------------------------------------------------+
+| ["2022-05-15 12:00:00", "2022-05-16 12:00:00"] |
++------------------------------------------------+
+
+mysql> select array_range(cast('2019-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 2 year) as array_range_2_year;
++------------------------------------------------+
+| array_range_2_year |
++------------------------------------------------+
+| ["2019-05-15 12:00:00", "2021-05-15 12:00:00"] |
++------------------------------------------------+
```
### keywords
diff --git a/docs/en/docs/sql-manual/sql-functions/array-functions/sequence.md b/docs/en/docs/sql-manual/sql-functions/array-functions/sequence.md
new file mode 100644
index 00000000000..b5e45413fa4
--- /dev/null
+++ b/docs/en/docs/sql-manual/sql-functions/array-functions/sequence.md
@@ -0,0 +1,104 @@
+---
+{
+ "title": "SEQUENCE",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## sequence
+
+<version since="dev">
+
+sequence
+
+</version>
+
+### description
+alias of array_range function
+
+#### Syntax
+
+```sql
+ARRAY<Int> sequence(Int end)
+ARRAY<Int> sequence(Int start, Int end)
+ARRAY<Int> sequence(Int start, Int end, Int step)
+ARRAY<Datetime> sequence(Datetime start_datetime, Datetime end_datetime)
+ARRAY<Datetime> sequence(Datetime start_datetime, Datetime end_datetime, INTERVAL Int interval_step UNIT)
+```
+1. To generate array of int:
+The parameters are all positive integers.
+start default value is 0, and step default value is 1.
+Return the array which numbers from start to end - 1 by step.
+
+2. To generate array of datetime:
+At least taking two parameters.
+The first two parameters are all datetimev2, the third is positive integer.
+If the third part is missing, `INTERVAL 1 DAY` will be default value.
+UNIT supports YEAR/MONTH/WEEK/DAY/HOUR/MINUTE/SECOND.
+Return the array of datetimev2 between start_datetime and closest to end_datetime by interval_step UNIT.
+
+### notice
+
+`if the 3rd parameter step/interval_step is negative or zero, the function will return NULL`
+
+### example
+
+```
+mysql> select sequence(10);
++--------------------------------+
+| sequence(10) |
++--------------------------------+
+| [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
++--------------------------------+
+
+mysql> select sequence(10,20);
++------------------------------------------+
+| sequence(10, 20) |
++------------------------------------------+
+| [10, 11, 12, 13, 14, 15, 16, 17, 18, 19] |
++------------------------------------------+
+
+mysql> select sequence(0,20,2);
++-------------------------------------+
+| sequence(0, 20, 2) |
++-------------------------------------+
+| [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] |
++-------------------------------------+
+
+mysql> select sequence(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0))) AS sequence_default;
++------------------------------------------------+
+| sequence_default |
++------------------------------------------------+
+| ["2022-05-15 12:00:00", "2022-05-16 12:00:00"] |
++------------------------------------------------+
+
+mysql> select sequence(cast('2019-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 2 year) as sequence_2_year;
++------------------------------------------------+
+| sequence_2_year |
++------------------------------------------------+
+| ["2019-05-15 12:00:00", "2021-05-15 12:00:00"] |
++------------------------------------------------+
+```
+
+### keywords
+
+ARRAY, RANGE, SEQUENCE
diff --git a/docs/sidebars.json b/docs/sidebars.json
index a01cb7e40d6..e21344bca35 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -354,7 +354,8 @@
"sql-manual/sql-functions/array-functions/arrays-overlap",
"sql-manual/sql-functions/array-functions/array-count",
"sql-manual/sql-functions/array-functions/countequal",
- "sql-manual/sql-functions/array-functions/element-at"
+ "sql-manual/sql-functions/array-functions/element-at",
+ "sql-manual/sql-functions/array-functions/sequence"
]
},
{
diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/array-range.md b/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/array-range.md
index cc68bf093d6..8e4b9148754 100644
--- a/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/array-range.md
+++ b/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/array-range.md
@@ -40,19 +40,26 @@ array_range
ARRAY<Int> array_range(Int end)
ARRAY<Int> array_range(Int start, Int end)
ARRAY<Int> array_range(Int start, Int end, Int step)
+ARRAY<Datetime> array_range(Datetime start_datetime, Datetime end_datetime)
+ARRAY<Datetime> array_range(Datetime start_datetime, Datetime end_datetime, INTERVAL Int interval_step UNIT)
```
+1. 生成int数组:
参数均为正整数 start 默认为 0, step 默认为 1。
最终返回一个数组,从start 到 end - 1, 步长为 step。
+2. 生成日期时间数组:
+至少取两个参数。
+前两个参数都是datetimev2,第三个是正整数。
+如果缺少第三部分,则`INTERVAL 1 DAY`将为默认值。
+UNIT 支持年/月/周/日/小时/分钟/秒。
+返回 start_datetime 和最接近 end_datetime 之间的 datetimev2 数组(按 Interval_step UNIT 计算)。
### notice
-`仅支持向量化引擎中使用`
+`如果第三个参数 step/interval_step 为负数或者零, 函数结果将为NULL`
### example
```
-mysql> set enable_vectorized_engine=true;
-
mysql> select array_range(10);
+--------------------------------+
| array_range(10) |
@@ -73,6 +80,20 @@ mysql> select array_range(0,20,2);
+-------------------------------------+
| [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] |
+-------------------------------------+
+
+mysql> select array_range(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0))) AS array_range_default;
++------------------------------------------------+
+| array_range_default |
++------------------------------------------------+
+| ["2022-05-15 12:00:00", "2022-05-16 12:00:00"] |
++------------------------------------------------+
+
+mysql> select array_range(cast('2019-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 2 year) as array_range_2_year;
++------------------------------------------------+
+| array_range_2_year |
++------------------------------------------------+
+| ["2019-05-15 12:00:00", "2021-05-15 12:00:00"] |
++------------------------------------------------+
```
### keywords
diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/sequence.md b/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/sequence.md
new file mode 100644
index 00000000000..69a5744b756
--- /dev/null
+++ b/docs/zh-CN/docs/sql-manual/sql-functions/array-functions/sequence.md
@@ -0,0 +1,102 @@
+---
+{
+ "title": "SEQUENCE",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## sequence
+
+<version since="dev">
+
+sequence
+
+</version>
+
+### description
+函数array_range的别称
+
+#### Syntax
+
+```sql
+ARRAY<Int> sequence(Int end)
+ARRAY<Int> sequence(Int start, Int end)
+ARRAY<Int> sequence(Int start, Int end, Int step)
+ARRAY<Datetime> sequence(Datetime start_datetime, Datetime end_datetime)
+ARRAY<Datetime> sequence(Datetime start_datetime, Datetime end_datetime, INTERVAL Int interval_step UNIT)
+```
+1. 生成int数组:
+参数均为正整数 start 默认为 0, step 默认为 1。
+最终返回一个数组,从start 到 end - 1, 步长为 step。
+2. 生成日期时间数组:
+至少取两个参数。
+前两个参数都是datetimev2,第三个是正整数。
+如果缺少第三部分,则`INTERVAL 1 DAY`将为默认值。
+UNIT 支持年/月/周/日/小时/分钟/秒。
+返回 start_datetime 和最接近 end_datetime 之间的 datetimev2 数组(按 Interval_step UNIT 计算)。
+
+### notice
+
+`如果第三个参数 step/interval_step 为负数或者零, 函数结果将为NULL`
+
+### example
+
+```
+mysql> select sequence(10);
++--------------------------------+
+| sequence(10) |
++--------------------------------+
+| [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] |
++--------------------------------+
+
+mysql> select sequence(10,20);
++------------------------------------------+
+| sequence(10, 20) |
++------------------------------------------+
+| [10, 11, 12, 13, 14, 15, 16, 17, 18, 19] |
++------------------------------------------+
+
+mysql> select sequence(0,20,2);
++-------------------------------------+
+| sequence(0, 20, 2) |
++-------------------------------------+
+| [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] |
++-------------------------------------+
+
+mysql> select sequence(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0))) AS sequence_default;
++------------------------------------------------+
+| sequence_default |
++------------------------------------------------+
+| ["2022-05-15 12:00:00", "2022-05-16 12:00:00"] |
++------------------------------------------------+
+
+mysql> select sequence(cast('2019-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 2 year) as sequence_2_year;
++------------------------------------------------+
+| sequence_2_year |
++------------------------------------------------+
+| ["2019-05-15 12:00:00", "2021-05-15 12:00:00"] |
++------------------------------------------------+
+```
+
+### keywords
+
+ARRAY, RANGE, SEQUENCE
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
index 7897232a953..2485f11eabe 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
@@ -108,6 +108,7 @@ AND: 'AND';
ANTI: 'ANTI';
APPEND: 'APPEND';
ARRAY: 'ARRAY';
+ARRAY_RANGE: 'ARRAY_RANGE';
AS: 'AS';
ASC: 'ASC';
AT: 'AT';
@@ -469,6 +470,7 @@ SCHEMAS: 'SCHEMAS';
SECOND: 'SECOND';
SELECT: 'SELECT';
SEMI: 'SEMI';
+SEQUENCE: 'SEQUENCE';
SERIALIZABLE: 'SERIALIZABLE';
SESSION: 'SESSION';
SET: 'SET';
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index cb34af20e62..dd638493720 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -674,6 +674,13 @@ primaryExpression
(INTERVAL unitsAmount=valueExpression unit=datetimeUnit
| unitsAmount=valueExpression)
RIGHT_PAREN #dateCeil
+ | name =(ARRAY_RANGE | SEQUENCE)
+ LEFT_PAREN
+ start=valueExpression COMMA
+ end=valueExpression COMMA
+ (INTERVAL unitsAmount=valueExpression unit=datetimeUnit
+ | unitsAmount=valueExpression)
+ RIGHT_PAREN #arrayRange
| name=CURRENT_DATE #currentDate
| name=CURRENT_TIME #currentTime
| name=CURRENT_TIMESTAMP #currentTimestamp
@@ -918,6 +925,7 @@ nonReserved
| ALIAS
| ANALYZED
| ARRAY
+ | ARRAY_RANGE
| AT
| AUTHORS
| AUTO_INCREMENT
@@ -1144,6 +1152,7 @@ nonReserved
| SCHEMA
| SECOND
| SERIALIZABLE
+ | SEQUENCE
| SESSION
| SHAPE
| SKEW
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index 1460adaf048..8d27ea06ea1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -1776,6 +1776,13 @@ public class FunctionCallExpr extends Expr {
+ "year|quarter|month|week|day|hour|minute|second");
}
}
+ if (fnName.getFunction().equalsIgnoreCase("array_range")
+ || fnName.getFunction().equalsIgnoreCase("sequence")) {
+ if (getChild(0) instanceof DateLiteral && !(getChild(2) instanceof StringLiteral)) {
+ throw new AnalysisException("To generate datetime array, please use interval literal like: "
+ + "interval 1 day.");
+ }
+ }
if (fnName.getFunction().equalsIgnoreCase("char")) {
if (!getChild(0).isConstant()) {
throw new AnalysisException(
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java
index f5928650efb..33f353503ab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java
@@ -485,7 +485,7 @@ public class BuiltinScalarFunctions implements FunctionHelper {
scalar(ArrayProduct.class, "array_product"),
scalar(ArrayPushBack.class, "Array_pushback"),
scalar(ArrayPushFront.class, "Array_pushfront"),
- scalar(ArrayRange.class, "array_range"),
+ scalar(ArrayRange.class, "array_range", "sequence"),
scalar(ArrayRemove.class, "array_remove"),
scalar(ArrayRepeat.class, "array_repeat"),
scalar(ArrayReverseSort.class, "array_reverse_sort"),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 9fd280c7f65..bbec47aaeef 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -48,6 +48,7 @@ import org.apache.doris.nereids.DorisParser.AlterMTMVContext;
import org.apache.doris.nereids.DorisParser.ArithmeticBinaryContext;
import org.apache.doris.nereids.DorisParser.ArithmeticUnaryContext;
import org.apache.doris.nereids.DorisParser.ArrayLiteralContext;
+import org.apache.doris.nereids.DorisParser.ArrayRangeContext;
import org.apache.doris.nereids.DorisParser.ArraySliceContext;
import org.apache.doris.nereids.DorisParser.BitOperationContext;
import org.apache.doris.nereids.DorisParser.BooleanExpressionContext;
@@ -261,6 +262,14 @@ import org.apache.doris.nereids.trees.expressions.functions.Function;
import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
import org.apache.doris.nereids.trees.expressions.functions.agg.GroupConcat;
import org.apache.doris.nereids.trees.expressions.functions.scalar.Array;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRange;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeDayUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeHourUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeMinuteUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeMonthUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeSecondUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeWeekUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeYearUnit;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArraySlice;
import org.apache.doris.nereids.trees.expressions.functions.scalar.Char;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ConvertTo;
@@ -1682,6 +1691,40 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
+ ", supported time unit: YEAR/MONTH/DAY/HOUR/MINUTE/SECOND", ctx);
}
+ @Override
+ public Expression visitArrayRange(ArrayRangeContext ctx) {
+ Expression start = (Expression) visit(ctx.start);
+ Expression end = (Expression) visit(ctx.end);
+ Expression step = (Expression) visit(ctx.unitsAmount);
+
+ String unit = ctx.unit.getText();
+ if (unit != null && !unit.isEmpty()) {
+ if ("Year".equalsIgnoreCase(unit)) {
+ return new ArrayRangeYearUnit(start, end, step);
+ } else if ("Month".equalsIgnoreCase(unit)) {
+ return new ArrayRangeMonthUnit(start, end, step);
+ } else if ("Week".equalsIgnoreCase(unit)) {
+ return new ArrayRangeWeekUnit(start, end, step);
+ } else if ("Day".equalsIgnoreCase(unit)) {
+ return new ArrayRangeDayUnit(start, end, step);
+ } else if ("Hour".equalsIgnoreCase(unit)) {
+ return new ArrayRangeHourUnit(start, end, step);
+ } else if ("Minute".equalsIgnoreCase(unit)) {
+ return new ArrayRangeMinuteUnit(start, end, step);
+ } else if ("Second".equalsIgnoreCase(unit)) {
+ return new ArrayRangeSecondUnit(start, end, step);
+ }
+ throw new ParseException("Unsupported time unit: " + ctx.unit
+ + ", supported time unit: YEAR/MONTH/DAY/HOUR/MINUTE/SECOND", ctx);
+ } else if (ctx.unitsAmount != null) {
+ return new ArrayRange(start, end, step);
+ } else if (ctx.end != null) {
+ return new ArrayRange(start, end);
+ } else {
+ return new ArrayRange(start);
+ }
+ }
+
@Override
public Expression visitDate_sub(Date_subContext ctx) {
Expression timeStamp = (Expression) visit(ctx.timestamp);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
index 70d0d4a0ef8..44da3b09b80 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
@@ -23,6 +23,7 @@ import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -40,7 +41,11 @@ public class ArrayRange extends ScalarFunction
FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+ .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE),
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT),
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
/**
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeDayUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeDayUnit.java
index 70d0d4a0ef8..c353ed06c87 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeDayUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_day_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeDayUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeDayUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_day_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeDayUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeDayUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeDayUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeHourUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeHourUnit.java
index 70d0d4a0ef8..320f7d76be4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeHourUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_hour_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeHourUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeHourUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_hour_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeHourUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeHourUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeHourUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeMinuteUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeMinuteUnit.java
index 70d0d4a0ef8..0a8f946f284 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeMinuteUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_minute_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeMinuteUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeMinuteUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_minute_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeMinuteUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeMinuteUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeMinuteUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeMonthUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeMonthUnit.java
index 70d0d4a0ef8..b3f4da2da82 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeMonthUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_month_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeMonthUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeMonthUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_month_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeMonthUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeMonthUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeMonthUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeSecondUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeSecondUnit.java
index 70d0d4a0ef8..7543be64df6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeSecondUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_second_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeSecondUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeSecondUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_second_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeSecondUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeSecondUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeSecondUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeWeekUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeWeekUnit.java
index 70d0d4a0ef8..de7c27bf720 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeWeekUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_week_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeWeekUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeWeekUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_week_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeWeekUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeWeekUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeWeekUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeYearUnit.java
similarity index 52%
copy from fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
copy to fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeYearUnit.java
index 70d0d4a0ef8..9cda8585b04 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRangeYearUnit.java
@@ -21,8 +21,10 @@ import org.apache.doris.catalog.FunctionSignature;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.BinaryExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.DateTimeV2Type;
import org.apache.doris.nereids.types.IntegerType;
import com.google.common.base.Preconditions;
@@ -31,63 +33,38 @@ import com.google.common.collect.ImmutableList;
import java.util.List;
/**
- * ScalarFunction 'array_range'. This class is generated by GenerateFunction.
+ * ScalarFunction 'array_range_year_unit'.
*/
-public class ArrayRange extends ScalarFunction
- implements ExplicitlyCastableSignature, AlwaysNullable {
-
- public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE)).args(IntegerType.INSTANCE, IntegerType.INSTANCE),
- FunctionSignature.ret(ArrayType.of(IntegerType.INSTANCE))
- .args(IntegerType.INSTANCE, IntegerType.INSTANCE, IntegerType.INSTANCE)
+public class ArrayRangeYearUnit extends ScalarFunction
+ implements BinaryExpression, ExplicitlyCastableSignature, AlwaysNullable {
+ private static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(ArrayType.of(DateTimeV2Type.SYSTEM_DEFAULT))
+ .args(DateTimeV2Type.SYSTEM_DEFAULT, DateTimeV2Type.SYSTEM_DEFAULT, IntegerType.INSTANCE)
);
- /**
- * constructor with 1 argument.
- */
- public ArrayRange(Expression arg) {
- super("array_range", arg);
- }
-
- /**
- * constructor with 2 arguments.
- */
- public ArrayRange(Expression arg0, Expression arg1) {
- super("array_range", arg0, arg1);
- }
-
/**
* constructor with 3 arguments.
*/
- public ArrayRange(Expression arg0, Expression arg1, Expression arg2) {
- super("array_range", arg0, arg1, arg2);
+ public ArrayRangeYearUnit(Expression arg0, Expression arg1, Expression arg2) {
+ super("array_range_year_unit", arg0, arg1, arg2);
}
/**
* withChildren.
*/
@Override
- public ArrayRange withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 1
- || children.size() == 2
- || children.size() == 3);
- if (children.size() == 1) {
- return new ArrayRange(children.get(0));
- } else if (children.size() == 2) {
- return new ArrayRange(children.get(0), children.get(1));
- } else {
- return new ArrayRange(children.get(0), children.get(1), children.get(2));
- }
+ public ArrayRangeYearUnit withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 3);
+ return new ArrayRangeYearUnit(children.get(0), children.get(1), children.get(2));
}
@Override
- public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
- return visitor.visitArrayRange(this, context);
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
}
@Override
- public List<FunctionSignature> getSignatures() {
- return SIGNATURES;
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitArrayRangeYearUnit(this, context);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/ScalarFunctionVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/ScalarFunctionVisitor.java
index 183b4a73dac..7bf020ad84d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/ScalarFunctionVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/ScalarFunctionVisitor.java
@@ -57,6 +57,13 @@ import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayProduct;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayPushBack;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayPushFront;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRange;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeDayUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeHourUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeMinuteUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeMonthUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeSecondUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeWeekUnit;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRangeYearUnit;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRemove;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayRepeat;
import org.apache.doris.nereids.trees.expressions.functions.scalar.ArrayReverseSort;
@@ -573,6 +580,34 @@ public interface ScalarFunctionVisitor<R, C> {
return visitScalarFunction(arrayRange, context);
}
+ default R visitArrayRangeDayUnit(ArrayRangeDayUnit arrayRangeDayUnit, C context) {
+ return visitScalarFunction(arrayRangeDayUnit, context);
+ }
+
+ default R visitArrayRangeHourUnit(ArrayRangeHourUnit arrayRangeHourUnit, C context) {
+ return visitScalarFunction(arrayRangeHourUnit, context);
+ }
+
+ default R visitArrayRangeMinuteUnit(ArrayRangeMinuteUnit arrayRangeMinuteUnit, C context) {
+ return visitScalarFunction(arrayRangeMinuteUnit, context);
+ }
+
+ default R visitArrayRangeMonthUnit(ArrayRangeMonthUnit arrayRangeMonthUnit, C context) {
+ return visitScalarFunction(arrayRangeMonthUnit, context);
+ }
+
+ default R visitArrayRangeSecondUnit(ArrayRangeSecondUnit arrayRangeSecondUnit, C context) {
+ return visitScalarFunction(arrayRangeSecondUnit, context);
+ }
+
+ default R visitArrayRangeWeekUnit(ArrayRangeWeekUnit arrayRangeWeekUnit, C context) {
+ return visitScalarFunction(arrayRangeWeekUnit, context);
+ }
+
+ default R visitArrayRangeYearUnit(ArrayRangeYearUnit arrayRangeYearUnit, C context) {
+ return visitScalarFunction(arrayRangeYearUnit, context);
+ }
+
default R visitArrayRemove(ArrayRemove arrayRemove, C context) {
return visitScalarFunction(arrayRemove, context);
}
diff --git a/gensrc/script/doris_builtins_functions.py b/gensrc/script/doris_builtins_functions.py
index 0ecb05612ac..0e96d3fa756 100644
--- a/gensrc/script/doris_builtins_functions.py
+++ b/gensrc/script/doris_builtins_functions.py
@@ -827,9 +827,11 @@ visible_functions = {
[['array_repeat'], 'ARRAY_VARCHAR', ['VARCHAR', 'BIGINT'], 'ALWAYS_NOT_NULLABLE'],
[['array_repeat'], 'ARRAY_STRING', ['STRING', 'BIGINT'], 'ALWAYS_NOT_NULLABLE'],
- [['array_range'], 'ARRAY_INT', ['INT'], 'ALWAYS_NULLABLE'],
- [['array_range'], 'ARRAY_INT', ['INT', 'INT'], 'ALWAYS_NULLABLE'],
- [['array_range'], 'ARRAY_INT', ['INT', 'INT', 'INT'], 'ALWAYS_NULLABLE'],
+ [['array_range', 'sequence'], 'ARRAY_INT', ['INT'], 'ALWAYS_NULLABLE'],
+ [['array_range', 'sequence'], 'ARRAY_INT', ['INT', 'INT'], 'ALWAYS_NULLABLE'],
+ [['array_range', 'sequence'], 'ARRAY_INT', ['INT', 'INT', 'INT'], 'ALWAYS_NULLABLE'],
+ [['array_range', 'sequence'], 'ARRAY_DATETIMEV2', ['DATETIMEV2', 'DATETIMEV2'], 'ALWAYS_NULLABLE'],
+ [['array_range', 'sequence'], 'ARRAY_DATETIMEV2', ['DATETIMEV2', 'DATETIMEV2', 'INT'], 'ALWAYS_NULLABLE'],
[['array_zip'], 'ARRAY', ['ARRAY<T>', '...'], '', ['T']],
diff --git a/regression-test/data/nereids_function_p0/scalar_function/Array.out b/regression-test/data/nereids_function_p0/scalar_function/Array.out
index 92cb7a24e66..0ac1f221443 100644
--- a/regression-test/data/nereids_function_p0/scalar_function/Array.out
+++ b/regression-test/data/nereids_function_p0/scalar_function/Array.out
@@ -14235,3 +14235,197 @@ true
-- !array_repeat4 --
[]
+-- !array_range_datetime1 --
+\N
+["2012-03-01 01:00:01", "2012-03-02 01:00:01"]
+["2012-03-02 02:01:02", "2012-03-04 02:01:02"]
+["2012-03-03 03:02:03", "2012-03-06 03:02:03"]
+["2012-03-04 04:03:04", "2012-03-08 04:03:04"]
+["2012-03-05 05:04:05", "2012-03-10 05:04:05"]
+["2012-03-06 06:05:06", "2012-03-12 06:05:06"]
+["2012-03-07 07:06:07", "2012-03-14 07:06:07"]
+["2012-03-08 08:07:08", "2012-03-16 08:07:08"]
+["2012-03-09 09:08:09", "2012-03-18 09:08:09"]
+["2012-03-10 10:09:10", "2012-03-20 10:09:10"]
+["2012-03-11 11:10:11", "2012-03-22 11:10:11"]
+["2012-03-12 12:11:12", "2012-03-24 12:11:12"]
+
+-- !array_range_datetime2 --
+\N
+["2012-03-01 01:00:01", "2012-03-08 01:00:01", "2012-03-15 01:00:01"]
+["2012-03-02 02:01:02", "2012-03-16 02:01:02"]
+["2012-03-03 03:02:03", "2012-03-24 03:02:03"]
+["2012-03-04 04:03:04", "2012-04-01 04:03:04"]
+["2012-03-05 05:04:05", "2012-04-09 05:04:05"]
+["2012-03-06 06:05:06", "2012-04-17 06:05:06"]
+["2012-03-07 07:06:07", "2012-04-25 07:06:07"]
+["2012-03-08 08:07:08", "2012-05-03 08:07:08"]
+["2012-03-09 09:08:09", "2012-05-11 09:08:09"]
+["2012-03-10 10:09:10", "2012-05-19 10:09:10"]
+["2012-03-11 11:10:11", "2012-05-27 11:10:11"]
+["2012-03-12 12:11:12", "2012-06-04 12:11:12"]
+
+-- !sequence_int_one_para --
+\N
+[0]
+[0, 1]
+[0, 1, 2]
+[0, 1, 2, 3]
+[0, 1, 2, 3, 4]
+[0, 1, 2, 3, 4, 5]
+[0, 1, 2, 3, 4, 5, 6]
+[0, 1, 2, 3, 4, 5, 6, 7]
+[0, 1, 2, 3, 4, 5, 6, 7, 8]
+[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
+[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
+
+-- !sequence_int_two_para --
+\N
+[1, 2]
+[2, 3]
+[3, 4]
+[4, 5]
+[5, 6]
+[6, 7]
+[7, 8]
+[8, 9]
+[9, 10]
+[10, 11]
+[11, 12]
+[12, 13]
+
+-- !sequence_int_three_para --
+\N
+[0, 1, 2]
+[1, 2, 3]
+[2, 3, 4]
+[3, 4, 5]
+[4, 5, 6]
+[5, 6, 7]
+[6, 7, 8]
+[7, 8, 9]
+[8, 9, 10]
+[9, 10, 11]
+[10, 11, 12]
+[11, 12, 13]
+
+-- !sequence_datetime_default --
+\N
+[]
+[]
+[]
+["2012-03-04 04:03:04"]
+["2012-03-05 05:04:05", "2012-03-06 05:04:05"]
+["2012-03-06 06:05:06", "2012-03-07 06:05:06", "2012-03-08 06:05:06"]
+["2012-03-07 07:06:07", "2012-03-08 07:06:07", "2012-03-09 07:06:07", "2012-03-10 07:06:07"]
+["2012-03-08 08:07:08", "2012-03-09 08:07:08", "2012-03-10 08:07:08", "2012-03-11 08:07:08", "2012-03-12 08:07:08"]
+["2012-03-09 09:08:09", "2012-03-10 09:08:09", "2012-03-11 09:08:09", "2012-03-12 09:08:09", "2012-03-13 09:08:09", "2012-03-14 09:08:09"]
+["2012-03-10 10:09:10", "2012-03-11 10:09:10", "2012-03-12 10:09:10", "2012-03-13 10:09:10", "2012-03-14 10:09:10", "2012-03-15 10:09:10", "2012-03-16 10:09:10"]
+["2012-03-11 11:10:11", "2012-03-12 11:10:11", "2012-03-13 11:10:11", "2012-03-14 11:10:11", "2012-03-15 11:10:11", "2012-03-16 11:10:11", "2012-03-17 11:10:11", "2012-03-18 11:10:11"]
+["2012-03-12 12:11:12", "2012-03-13 12:11:12", "2012-03-14 12:11:12", "2012-03-15 12:11:12", "2012-03-16 12:11:12", "2012-03-17 12:11:12", "2012-03-18 12:11:12", "2012-03-19 12:11:12", "2012-03-20 12:11:12"]
+
+-- !sequence_datetime_day --
+\N
+["2012-03-01 01:00:01", "2012-03-02 01:00:01"]
+["2012-03-02 02:01:02", "2012-03-04 02:01:02"]
+["2012-03-03 03:02:03", "2012-03-06 03:02:03"]
+["2012-03-04 04:03:04", "2012-03-08 04:03:04"]
+["2012-03-05 05:04:05", "2012-03-10 05:04:05"]
+["2012-03-06 06:05:06", "2012-03-12 06:05:06"]
+["2012-03-07 07:06:07", "2012-03-14 07:06:07"]
+["2012-03-08 08:07:08", "2012-03-16 08:07:08"]
+["2012-03-09 09:08:09", "2012-03-18 09:08:09"]
+["2012-03-10 10:09:10", "2012-03-20 10:09:10"]
+["2012-03-11 11:10:11", "2012-03-22 11:10:11"]
+["2012-03-12 12:11:12", "2012-03-24 12:11:12"]
+
+-- !sequence_datetime_week --
+\N
+["2012-03-01 01:00:01", "2012-03-08 01:00:01", "2012-03-15 01:00:01"]
+["2012-03-02 02:01:02", "2012-03-16 02:01:02"]
+["2012-03-03 03:02:03", "2012-03-24 03:02:03"]
+["2012-03-04 04:03:04", "2012-04-01 04:03:04"]
+["2012-03-05 05:04:05", "2012-04-09 05:04:05"]
+["2012-03-06 06:05:06", "2012-04-17 06:05:06"]
+["2012-03-07 07:06:07", "2012-04-25 07:06:07"]
+["2012-03-08 08:07:08", "2012-05-03 08:07:08"]
+["2012-03-09 09:08:09", "2012-05-11 09:08:09"]
+["2012-03-10 10:09:10", "2012-05-19 10:09:10"]
+["2012-03-11 11:10:11", "2012-05-27 11:10:11"]
+["2012-03-12 12:11:12", "2012-06-04 12:11:12"]
+
+-- !sequence_datetime_month --
+\N
+["2012-03-01 01:00:01", "2012-04-01 01:00:01", "2012-05-01 01:00:01", "2012-06-01 01:00:01"]
+["2012-03-02 02:01:02", "2012-05-02 02:01:02", "2012-07-02 02:01:02"]
+["2012-03-03 03:02:03", "2012-06-03 03:02:03"]
+["2012-03-04 04:03:04", "2012-07-04 04:03:04"]
+["2012-03-05 05:04:05", "2012-08-05 05:04:05"]
+["2012-03-06 06:05:06", "2012-09-06 06:05:06"]
+["2012-03-07 07:06:07", "2012-10-07 07:06:07"]
+["2012-03-08 08:07:08", "2012-11-08 08:07:08"]
+["2012-03-09 09:08:09", "2012-12-09 09:08:09"]
+["2012-03-10 10:09:10", "2013-01-10 10:09:10"]
+["2012-03-11 11:10:11", "2013-02-11 11:10:11"]
+["2012-03-12 12:11:12", "2013-03-12 12:11:12"]
+
+-- !sequence_datetime_year --
+\N
+["2012-03-01 01:00:01", "2013-03-01 01:00:01", "2014-03-01 01:00:01", "2015-03-01 01:00:01"]
+["2012-03-02 02:01:02", "2014-03-02 02:01:02", "2016-03-02 02:01:02"]
+["2012-03-03 03:02:03", "2015-03-03 03:02:03"]
+["2012-03-04 04:03:04", "2016-03-04 04:03:04"]
+["2012-03-05 05:04:05", "2017-03-05 05:04:05"]
+["2012-03-06 06:05:06", "2018-03-06 06:05:06"]
+["2012-03-07 07:06:07", "2019-03-07 07:06:07"]
+["2012-03-08 08:07:08", "2020-03-08 08:07:08"]
+["2012-03-09 09:08:09", "2021-03-09 09:08:09"]
+["2012-03-10 10:09:10", "2022-03-10 10:09:10"]
+["2012-03-11 11:10:11", "2023-03-11 11:10:11"]
+["2012-03-12 12:11:12", "2024-03-12 12:11:12"]
+
+-- !sequence_datetime_hour --
+\N
+[]
+[]
+[]
+["2012-03-04 04:03:04"]
+["2012-03-05 05:04:05"]
+["2012-03-06 06:05:06"]
+["2012-03-07 07:06:07"]
+["2012-03-08 08:07:08"]
+["2012-03-09 09:08:09"]
+["2012-03-10 10:09:10"]
+["2012-03-11 11:10:11"]
+["2012-03-12 12:11:12"]
+
+-- !sequence_datetime_minute --
+\N
+["2012-03-01 01:00:01", "2012-03-01 01:01:01"]
+["2012-03-02 02:01:02", "2012-03-02 02:03:02"]
+["2012-03-03 03:02:03", "2012-03-03 03:05:03"]
+["2012-03-04 04:03:04", "2012-03-04 04:07:04"]
+["2012-03-05 05:04:05", "2012-03-05 05:09:05"]
+["2012-03-06 06:05:06", "2012-03-06 06:11:06"]
+["2012-03-07 07:06:07", "2012-03-07 07:13:07"]
+["2012-03-08 08:07:08", "2012-03-08 08:15:08"]
+["2012-03-09 09:08:09", "2012-03-09 09:17:09"]
+["2012-03-10 10:09:10", "2012-03-10 10:19:10"]
+["2012-03-11 11:10:11", "2012-03-11 11:21:11"]
+["2012-03-12 12:11:12", "2012-03-12 12:23:12"]
+
+-- !sequence_datetime_second --
+\N
+\N
+["2012-03-02 02:01:02", "2012-03-02 02:01:03"]
+["2012-03-03 03:02:03", "2012-03-03 03:02:05"]
+["2012-03-04 04:03:04", "2012-03-04 04:03:07"]
+["2012-03-05 05:04:05", "2012-03-05 05:04:09"]
+["2012-03-06 06:05:06", "2012-03-06 06:05:11"]
+["2012-03-07 07:06:07", "2012-03-07 07:06:13"]
+["2012-03-08 08:07:08", "2012-03-08 08:07:15"]
+["2012-03-09 09:08:09", "2012-03-09 09:08:17"]
+["2012-03-10 10:09:10", "2012-03-10 10:09:19"]
+["2012-03-11 11:10:11", "2012-03-11 11:10:21"]
+["2012-03-12 12:11:12", "2012-03-12 12:11:23"]
\ No newline at end of file
diff --git a/regression-test/data/query_p0/sql_functions/array_functions/test_array_functions.out b/regression-test/data/query_p0/sql_functions/array_functions/test_array_functions.out
index 1b60f427058..5324b07bf64 100644
--- a/regression-test/data/query_p0/sql_functions/array_functions/test_array_functions.out
+++ b/regression-test/data/query_p0/sql_functions/array_functions/test_array_functions.out
@@ -1693,3 +1693,134 @@
1 [1.100000000, 1.100000000] 1.100
1 [1.200000000, 1.200000000] 1.200
+-- !table_select --
+-1 \N
+1 [0]
+2 [0, 1]
+3 [0, 1, 2]
+4 [0, 1, 2, 3]
+5 [0, 1, 2, 3, 4]
+6 [0, 1, 2, 3, 4, 5]
+7 [0, 1, 2, 3, 4, 5, 6]
+\N \N
+8 [0, 1, 2, 3, 4, 5, 6, 7]
+9 [0, 1, 2, 3, 4, 5, 6, 7, 8]
+
+-- !table_select --
+-1 \N
+1 [1, 2]
+2 [2, 3, 4, 5, 6, 7, 8, 9]
+3 \N
+4 [4, 5]
+5 [5, 6, 7, 8, 9]
+6 \N
+7 [7, 8, 9]
+\N \N
+8 []
+9 [9]
+
+-- !table_select --
+-1 \N
+1 [1]
+2 [2, 4, 6, 8]
+3 \N
+4 [4, 5]
+5 [5, 6, 7, 8, 9]
+6 \N
+7 \N
+\N \N
+8 []
+9 [9]
+
+-- !table_select --
+2022-05-15T12:00 2022-05-18T12:00:00.123 ["2022-05-15 12:00:00", "2022-05-16 12:00:00", "2022-05-17 12:00:00"]
+
+-- !table_select --
+2022-05-15T12:00 2022-05-18T12:00:00.123456 ["2022-05-15 12:00:00", "2022-05-16 12:00:00", "2022-05-17 12:00:00"]
+
+-- !table_select --
+2022-05-15T12:00 2022-05-18T12:00:00.123 1 ["2022-05-15 12:00:00", "2022-05-16 12:00:00", "2022-05-17 12:00:00"]
+
+-- !table_select --
+2022-04-22T12:00:00.123 2022-05-08T12:00:00.123456 1 ["2022-04-22 12:00:00", "2022-04-29 12:00:00"]
+
+-- !table_select --
+2022-01-15T12:00:00.123 2022-05-18T12:00:00.123456 2 ["2022-01-15 12:00:00", "2022-03-15 12:00:00"]
+
+-- !table_select --
+2015-05-15T12:00:00.123 2022-05-18T12:00:00.123456 3 ["2015-05-15 12:00:00", "2018-05-15 12:00:00", "2021-05-15 12:00:00"]
+
+-- !table_select --
+2022-05-18T12:00:00.123 2022-05-18T23:10:00.123456 4 ["2022-05-18 12:00:00", "2022-05-18 16:00:00", "2022-05-18 20:00:00"]
+
+-- !table_select --
+2022-05-18T12:00:00.123 2022-05-18T12:16 5 ["2022-05-18 12:00:00", "2022-05-18 12:05:00", "2022-05-18 12:10:00", "2022-05-18 12:15:00"]
+
+-- !table_select --
+2022-05-18T12:00:10 2022-05-18T12:00:30 6 ["2022-05-18 12:00:10", "2022-05-18 12:00:16", "2022-05-18 12:00:22", "2022-05-18 12:00:28"]
+
+-- !table_select --
+2022-05-18T12:00:10 2022-05-21T12:00:30 2 ["2022-05-18 12:00:10", "2022-05-20 12:00:10"]
+2022-05-22T12:00:10 2022-05-25T12:00:30 2 ["2022-05-22 12:00:10", "2022-05-24 12:00:10"]
+2022-05-23T12:00:10 2022-05-26T12:00:30 2 ["2022-05-23 12:00:10", "2022-05-25 12:00:10"]
+2022-05-27T12:00:10 2022-05-30T12:00:30 2 ["2022-05-27 12:00:10", "2022-05-29 12:00:10"]
+
+-- !const_select --
+["2022-05-15 12:00:00", "2022-05-16 12:00:00"]
+
+-- !const_select --
+["2022-05-15 12:00:00", "2022-05-16 12:00:00"]
+
+-- !const_select --
+[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
+
+-- !const_select --
+[3, 4, 5, 6, 7, 8, 9]
+
+-- !const_select --
+[3, 5, 7, 9]
+
+-- !const_select --
+["2022-05-15 12:00:00", "2022-05-16 12:00:00"]
+
+-- !const_select --
+["2022-05-15 12:00:00", "2022-05-16 12:00:00"]
+
+-- !const_select --
+["2022-05-01 12:00:00"]
+
+-- !const_select --
+["2022-01-15 12:00:00", "2022-04-15 12:00:00"]
+
+-- !const_select --
+["2019-05-15 12:00:00", "2021-05-15 12:00:00"]
+
+-- !const_select --
+["2022-05-18 12:00:10", "2022-05-18 16:00:10", "2022-05-18 20:00:10"]
+
+-- !const_select --
+["2022-05-18 12:00:10", "2022-05-18 12:05:10", "2022-05-18 12:10:10", "2022-05-18 12:15:10"]
+
+-- !const_select --
+["2022-05-18 12:00:10", "2022-05-18 12:00:16", "2022-05-18 12:00:22", "2022-05-18 12:00:28"]
+
+-- !const_select --
+\N
+
+-- !const_select --
+\N
+
+-- !const_select --
+\N
+
+-- !const_select --
+\N
+
+-- !const_select --
+\N
+
+-- !const_select --
+\N
+
+-- !const_select --
+\N
\ No newline at end of file
diff --git a/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy b/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
index 42b5cf8525c..e1b7d5d8d4b 100644
--- a/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
+++ b/regression-test/suites/nereids_function_p0/scalar_function/Array.groovy
@@ -1260,4 +1260,18 @@ suite("nereids_scalar_fn_Array") {
// array_zip
sql "select array_zip([1], ['1'], [1.0])"
+ // array_range with datetime argument, sequence with int and datetime argument
+ qt_array_range_datetime1 """select array_range(kdtmv2s1, date_add(kdtmv2s1, interval kint+1 day), interval kint day) from fn_test order by kdtmv2s1;"""
+ qt_array_range_datetime2 """select array_range(kdtmv2s1, date_add(kdtmv2s1, interval kint+2 week), interval kint week) from fn_test order by kdtmv2s1;"""
+ qt_sequence_int_one_para """select sequence(kint) from fn_test order by kint;"""
+ qt_sequence_int_two_para """select sequence(kint, kint+2) from fn_test order by kint;"""
+ qt_sequence_int_three_para """select sequence(kint-1, kint+2, 1) from fn_test order by kint;"""
+ qt_sequence_datetime_default """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint-3 day)) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_day """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint+1 day), interval kint day) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_week """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint+2 week), interval kint week) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_month """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint+3 month), interval kint month) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_year """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint+3 year), interval kint year) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_hour """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint-3 hour), interval kint hour) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_minute """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint+1 minute), interval kint minute) from fn_test order by kdtmv2s1;"""
+ qt_sequence_datetime_second """select sequence(kdtmv2s1, date_add(kdtmv2s1, interval kint second), interval kint-1 second) from fn_test order by kdtmv2s1;"""
}
\ No newline at end of file
diff --git a/regression-test/suites/query_p0/sql_functions/array_functions/test_array_functions.groovy b/regression-test/suites/query_p0/sql_functions/array_functions/test_array_functions.groovy
index 858a4bdbd37..99bb879e676 100644
--- a/regression-test/suites/query_p0/sql_functions/array_functions/test_array_functions.groovy
+++ b/regression-test/suites/query_p0/sql_functions/array_functions/test_array_functions.groovy
@@ -335,4 +335,88 @@ suite("test_array_functions") {
qt_sql """ select array_position(kadcml, kdcmls1), kadcml, kdcmls1 from fn_test;"""
+ /*
+ test scope:
+ 1.array_range function with datetimev2 type
+ 2.sequence function(alias of array_range) with int and datetimev2 type
+ */
+ sql "SET enable_nereids_planner=true"
+ def tableName5 = "tbl_test_sequence"
+ sql """drop TABLE if EXISTS ${tableName5};"""
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tableName5} (
+ `test_id` int(11) NULL COMMENT "",
+ `k1` int(11) NULL COMMENT "",
+ `k2` int(11) NULL COMMENT "",
+ `k3` int(11) NULL COMMENT "",
+ `k4` datetimev2(0) NULL COMMENT "",
+ `k5` datetimev2(3) NULL COMMENT "",
+ `k6` datetimev2(6) NULL COMMENT "",
+ `step` int(11) NULL COMMENT ""
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`test_id`)
+ DISTRIBUTED BY HASH(`test_id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "storage_format" = "V2"
+ );
+ """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(1,-1,3,5); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(2,1,3,5); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(3,2,10,2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(4,3,NULL,NULL); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(5,4,6,1); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(6,5,10,1); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(7,6,NULL,1); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(8,7,10,NULL); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(9,NULL,10,2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(10,8,2,2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k1, k2, k3) VALUES(11,9,10,6); """
+ sql """ INSERT INTO ${tableName5} (test_id, k4, k5) VALUES(12, '2022-05-15 12:00:00', '2022-05-18 12:00:00.123'); """
+ sql """ INSERT INTO ${tableName5} (test_id, k4, k6) VALUES(13, '2022-05-15 12:00:00', '2022-05-18 12:00:00.123456'); """
+ sql """ INSERT INTO ${tableName5} (test_id, k4, k5, step) VALUES(14, '2022-05-15 12:00:00', '2022-05-18 12:00:00.123', 1); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(15, '2022-04-22 12:00:00.123', '2022-05-08 12:00:00.123456', 1); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(16, '2022-01-15 12:00:00.123', '2022-05-18 12:00:00.123456', 2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(17, '2015-05-15 12:00:00.123', '2022-05-18 12:00:00.123456', 3); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(18, '2022-05-18 12:00:00.123', '2022-05-18 23:10:00.123456', 4); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(19, '2022-05-18 12:00:00.123', '2022-05-18 12:16:00', 5); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(20, '2022-05-18 12:00:10', '2022-05-18 12:00:30', 6); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(21, '2022-05-18 12:00:10', '2022-05-21 12:00:30', 2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(22, '2022-05-22 12:00:10', '2022-05-25 12:00:30', 2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(23, '2022-05-23 12:00:10', '2022-05-26 12:00:30', 2); """
+ sql """ INSERT INTO ${tableName5} (test_id, k5, k6, step) VALUES(24, '2022-05-27 12:00:10', '2022-05-30 12:00:30', 2); """
+
+ qt_table_select "SELECT k1, sequence(k1) from ${tableName5} where test_id < 12 ORDER BY test_id; "
+ qt_table_select "SELECT k1, sequence(k1,k2) from ${tableName5} where test_id < 12 ORDER BY test_id; "
+ qt_table_select "SELECT k1, sequence(k1,k2,k3) from ${tableName5} where test_id < 12 ORDER BY test_id; "
+ qt_table_select "SELECT k4, k5, sequence(k4, k5) from ${tableName5} where test_id = 12; "
+ qt_table_select "SELECT k4, k6, sequence(k4, k6) from ${tableName5} where test_id = 13; "
+ qt_table_select "SELECT k4, k5, step, sequence(k4, k5, interval step day) from ${tableName5} where test_id = 14; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step week) from ${tableName5} where test_id = 15; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step month) from ${tableName5} where test_id = 16; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step year) from ${tableName5} where test_id = 17; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step hour) from ${tableName5} where test_id = 18; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step minute) from ${tableName5} where test_id = 19; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step second) from ${tableName5} where test_id = 20; "
+ qt_table_select "SELECT k5, k6, step, sequence(k5, k6, interval step day) from ${tableName5} where test_id between 21 and 24 order by test_id; "
+ qt_const_select "select array_range(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0))); "
+ qt_const_select "select array_range(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 1 day); "
+ qt_const_select "select sequence(10); "
+ qt_const_select "select sequence(3, 10); "
+ qt_const_select "select sequence(3, 10, 2); "
+ qt_const_select "select sequence(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0))); "
+ qt_const_select "select sequence(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 1 day); "
+ qt_const_select "select sequence(cast('2022-05-01 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 2 week); "
+ qt_const_select "select sequence(cast('2022-01-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 3 month); "
+ qt_const_select "select sequence(cast('2019-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 2 year); "
+ qt_const_select "select sequence(cast('2022-05-18 12:00:10' as datetimev2(0)), cast('2022-05-18 22:00:30' as datetimev2(0)), interval 4 hour); "
+ qt_const_select "select sequence(cast('2022-05-18 12:00:10' as datetimev2(0)), cast('2022-05-18 12:16:30' as datetimev2(0)), interval 5 minute); "
+ qt_const_select "select sequence(cast('2022-05-18 12:00:10' as datetimev2(0)), cast('2022-05-18 12:00:30' as datetimev2(0)), interval 6 second); "
+ qt_const_select "select sequence(cast('2022-05-18 12:00:10' as datetimev2(0)), cast('2022-05-18 22:00:30' as datetimev2(0)), interval -4 hour); "
+ qt_const_select "select sequence(cast('2022-35-38 12:00:10' as datetimev2(0)), cast('2022-05-18 22:00:30' as datetimev2(0)), interval 4 hour); "
+ qt_const_select "select sequence(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-35-37 12:00:00' as datetimev2(0))); "
+ qt_const_select "select sequence(1, 10, interval 10 day); "
+ qt_const_select "select sequence(cast('2022-35-38 12:00:10' as datetimev2(0)), cast('2022-05-18 22:00:30' as datetimev2(0))); "
+ qt_const_select "select sequence(1, 10, 0); "
+ qt_const_select "select sequence(cast('2022-05-15 12:00:00' as datetimev2(0)), cast('2022-05-17 12:00:00' as datetimev2(0)), interval 0 day); "
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 06/06: [Improvement](executor)add remote scan thread pool (#31376)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 743eb45695d6bcc232eb63d4023d7bb6109f2edc
Author: wangbo <wa...@apache.org>
AuthorDate: Mon Feb 26 19:01:46 2024 +0800
[Improvement](executor)add remote scan thread pool (#31376)
* add remote scan thread pool
* +1
---
be/src/agent/cgroup_cpu_ctl.cpp | 2 +-
be/src/agent/cgroup_cpu_ctl.h | 5 +-
be/src/agent/workload_group_listener.cpp | 2 +-
be/src/runtime/query_context.cpp | 4 +-
be/src/runtime/query_context.h | 5 +
be/src/runtime/task_group/task_group.cpp | 172 ++++++++++++++++++++-
be/src/runtime/task_group/task_group.h | 28 ++++
be/src/runtime/task_group/task_group_manager.cpp | 171 +-------------------
be/src/runtime/task_group/task_group_manager.h | 23 +--
be/src/vec/exec/scan/scanner_context.cpp | 1 +
be/src/vec/exec/scan/scanner_context.h | 3 +
be/src/vec/exec/scan/scanner_scheduler.cpp | 24 ++-
be/src/vec/exec/scan/scanner_scheduler.h | 34 +++-
.../resource/workloadgroup/WorkloadGroup.java | 47 +++++-
.../resource/workloadgroup/WorkloadGroupMgr.java | 3 +-
.../doris/tablefunction/MetadataGenerator.java | 10 +-
.../WorkloadGroupsTableValuedFunction.java | 2 +
gensrc/thrift/BackendService.thrift | 2 +
gensrc/thrift/FrontendService.thrift | 2 +-
19 files changed, 322 insertions(+), 218 deletions(-)
diff --git a/be/src/agent/cgroup_cpu_ctl.cpp b/be/src/agent/cgroup_cpu_ctl.cpp
index c94a3c05f1e..5263c060530 100644
--- a/be/src/agent/cgroup_cpu_ctl.cpp
+++ b/be/src/agent/cgroup_cpu_ctl.cpp
@@ -172,7 +172,7 @@ Status CgroupV1CpuCtl::modify_cg_cpu_soft_limit_no_lock(int cpu_shares) {
Status CgroupV1CpuCtl::modify_cg_cpu_hard_limit_no_lock(int cpu_hard_limit) {
int val = cpu_hard_limit > 0 ? (_cpu_cfs_period_us * _cpu_core_num * cpu_hard_limit / 100)
- : CPU_HARD_LIMIT_DEFAULT_VALUE;
+ : CGROUP_CPU_HARD_LIMIT_DEFAULT_VALUE;
std::string msg = "modify cpu quota value to " + std::to_string(val);
return CgroupCpuCtl::write_cg_sys_file(_cgroup_v1_cpu_tg_quota_file, val, msg, false);
}
diff --git a/be/src/agent/cgroup_cpu_ctl.h b/be/src/agent/cgroup_cpu_ctl.h
index 94514c8e2e0..1289f26307b 100644
--- a/be/src/agent/cgroup_cpu_ctl.h
+++ b/be/src/agent/cgroup_cpu_ctl.h
@@ -29,10 +29,7 @@
namespace doris {
// cgroup cpu.cfs_quota_us default value, it means disable cpu hard limit
-const static int CPU_HARD_LIMIT_DEFAULT_VALUE = -1;
-
-// cgroup cpu.shares default value
-const static uint64_t CPU_SOFT_LIMIT_DEFAULT_VALUE = 1024;
+const static int CGROUP_CPU_HARD_LIMIT_DEFAULT_VALUE = -1;
class CgroupCpuCtl {
public:
diff --git a/be/src/agent/workload_group_listener.cpp b/be/src/agent/workload_group_listener.cpp
index 5f9da64bd2a..1d5a8544e11 100644
--- a/be/src/agent/workload_group_listener.cpp
+++ b/be/src/agent/workload_group_listener.cpp
@@ -50,7 +50,7 @@ void WorkloadGroupListener::handle_topic_info(const std::vector<TopicInfo>& topi
task_group_info.enable_cpu_hard_limit);
// 4 create and update task scheduler
- _exec_env->task_group_manager()->upsert_cg_task_scheduler(&task_group_info, _exec_env);
+ tg->upsert_task_scheduler(&task_group_info, _exec_env);
LOG(INFO) << "update task group finish, tg info=" << tg->debug_string()
<< ", enable_cpu_hard_limit="
diff --git a/be/src/runtime/query_context.cpp b/be/src/runtime/query_context.cpp
index dadb6ada172..2c25d37d14b 100644
--- a/be/src/runtime/query_context.cpp
+++ b/be/src/runtime/query_context.cpp
@@ -210,8 +210,8 @@ Status QueryContext::set_task_group(taskgroup::TaskGroupPtr& tg) {
// see task_group_manager::delete_task_group_by_ids
RETURN_IF_ERROR(_task_group->add_query(_query_id));
_task_group->add_mem_tracker_limiter(query_mem_tracker);
- _exec_env->task_group_manager()->get_query_scheduler(
- _task_group->id(), &_task_scheduler, &_scan_task_scheduler, &_non_pipe_thread_pool);
+ _task_group->get_query_scheduler(&_task_scheduler, &_scan_task_scheduler,
+ &_non_pipe_thread_pool, &_remote_scan_task_scheduler);
return Status::OK();
}
diff --git a/be/src/runtime/query_context.h b/be/src/runtime/query_context.h
index d7b3813dcef..a639268c552 100644
--- a/be/src/runtime/query_context.h
+++ b/be/src/runtime/query_context.h
@@ -196,6 +196,10 @@ public:
vectorized::SimplifiedScanScheduler* get_scan_scheduler() { return _scan_task_scheduler; }
+ vectorized::SimplifiedScanScheduler* get_remote_scan_scheduler() {
+ return _remote_scan_task_scheduler;
+ }
+
pipeline::Dependency* get_execution_dependency() { return _execution_dependency.get(); }
void register_query_statistics(std::shared_ptr<QueryStatistics> qs);
@@ -283,6 +287,7 @@ private:
doris::pipeline::TaskScheduler* _task_scheduler = nullptr;
vectorized::SimplifiedScanScheduler* _scan_task_scheduler = nullptr;
ThreadPool* _non_pipe_thread_pool = nullptr;
+ vectorized::SimplifiedScanScheduler* _remote_scan_task_scheduler = nullptr;
std::unique_ptr<pipeline::Dependency> _execution_dependency;
std::shared_ptr<QueryStatistics> _cpu_statistics = nullptr;
diff --git a/be/src/runtime/task_group/task_group.cpp b/be/src/runtime/task_group/task_group.cpp
index ddddf39dbc8..53534444a5d 100644
--- a/be/src/runtime/task_group/task_group.cpp
+++ b/be/src/runtime/task_group/task_group.cpp
@@ -34,6 +34,7 @@
#include "util/mem_info.h"
#include "util/parse_util.h"
#include "util/runtime_profile.h"
+#include "util/threadpool.h"
#include "vec/exec/scan/scanner_scheduler.h"
namespace doris {
@@ -43,6 +44,7 @@ const static uint64_t CPU_SHARE_DEFAULT_VALUE = 1024;
const static std::string MEMORY_LIMIT_DEFAULT_VALUE = "0%";
const static bool ENABLE_MEMORY_OVERCOMMIT_DEFAULT_VALUE = true;
const static int CPU_HARD_LIMIT_DEFAULT_VALUE = -1;
+const static uint64_t CPU_SOFT_LIMIT_DEFAULT_VALUE = 1024;
TaskGroup::TaskGroup(const TaskGroupInfo& tg_info)
: _id(tg_info.id),
@@ -53,16 +55,19 @@ TaskGroup::TaskGroup(const TaskGroupInfo& tg_info)
_cpu_share(tg_info.cpu_share),
_mem_tracker_limiter_pool(MEM_TRACKER_GROUP_NUM),
_cpu_hard_limit(tg_info.cpu_hard_limit),
- _scan_thread_num(tg_info.scan_thread_num) {}
+ _scan_thread_num(tg_info.scan_thread_num),
+ _max_remote_scan_thread_num(tg_info.max_remote_scan_thread_num),
+ _min_remote_scan_thread_num(tg_info.min_remote_scan_thread_num) {}
std::string TaskGroup::debug_string() const {
std::shared_lock<std::shared_mutex> rl {_mutex};
return fmt::format(
"TG[id = {}, name = {}, cpu_share = {}, memory_limit = {}, enable_memory_overcommit = "
- "{}, version = {}, cpu_hard_limit = {}, scan_thread_num = {}]",
+ "{}, version = {}, cpu_hard_limit = {}, scan_thread_num = "
+ "{}, max_remote_scan_thread_num = {}, min_remote_scan_thread_num = {}]",
_id, _name, cpu_share(), PrettyPrinter::print(_memory_limit, TUnit::BYTES),
_enable_memory_overcommit ? "true" : "false", _version, cpu_hard_limit(),
- _scan_thread_num);
+ _scan_thread_num, _max_remote_scan_thread_num, _min_remote_scan_thread_num);
}
void TaskGroup::check_and_update(const TaskGroupInfo& tg_info) {
@@ -85,6 +90,8 @@ void TaskGroup::check_and_update(const TaskGroupInfo& tg_info) {
_cpu_share = tg_info.cpu_share;
_cpu_hard_limit = tg_info.cpu_hard_limit;
_scan_thread_num = tg_info.scan_thread_num;
+ _max_remote_scan_thread_num = tg_info.max_remote_scan_thread_num;
+ _min_remote_scan_thread_num = tg_info.min_remote_scan_thread_num;
} else {
return;
}
@@ -266,8 +273,167 @@ Status TaskGroupInfo::parse_topic_info(const TWorkloadGroupInfo& workload_group_
task_group_info->scan_thread_num = workload_group_info.scan_thread_num;
}
+ // 10 max remote scan thread num
+ task_group_info->max_remote_scan_thread_num = config::doris_scanner_thread_pool_thread_num;
+ if (workload_group_info.__isset.max_remote_scan_thread_num &&
+ workload_group_info.max_remote_scan_thread_num > 0) {
+ task_group_info->max_remote_scan_thread_num =
+ workload_group_info.max_remote_scan_thread_num;
+ }
+
+ // 11 min remote scan thread num
+ task_group_info->min_remote_scan_thread_num = config::doris_scanner_thread_pool_thread_num;
+ if (workload_group_info.__isset.min_remote_scan_thread_num &&
+ workload_group_info.min_remote_scan_thread_num > 0) {
+ task_group_info->min_remote_scan_thread_num =
+ workload_group_info.min_remote_scan_thread_num;
+ }
+
return Status::OK();
}
+void TaskGroup::upsert_task_scheduler(taskgroup::TaskGroupInfo* tg_info, ExecEnv* exec_env) {
+ uint64_t tg_id = tg_info->id;
+ std::string tg_name = tg_info->name;
+ int cpu_hard_limit = tg_info->cpu_hard_limit;
+ uint64_t cpu_shares = tg_info->cpu_share;
+ bool enable_cpu_hard_limit = tg_info->enable_cpu_hard_limit;
+ int scan_thread_num = tg_info->scan_thread_num;
+ int max_remote_scan_thread_num = tg_info->max_remote_scan_thread_num;
+ int min_remote_scan_thread_num = tg_info->min_remote_scan_thread_num;
+
+ std::lock_guard<std::shared_mutex> wlock(_task_sched_lock);
+ if (config::doris_cgroup_cpu_path != "" && _cgroup_cpu_ctl == nullptr) {
+ std::unique_ptr<CgroupCpuCtl> cgroup_cpu_ctl = std::make_unique<CgroupV1CpuCtl>(tg_id);
+ Status ret = cgroup_cpu_ctl->init();
+ if (ret.ok()) {
+ _cgroup_cpu_ctl = std::move(cgroup_cpu_ctl);
+ LOG(INFO) << "[upsert wg thread pool] cgroup init success";
+ } else {
+ LOG(INFO) << "[upsert wg thread pool] cgroup init failed, gid= " << tg_id
+ << ", reason=" << ret.to_string();
+ }
+ }
+
+ CgroupCpuCtl* cg_cpu_ctl_ptr = _cgroup_cpu_ctl.get();
+
+ if (_task_sched == nullptr) {
+ int32_t executors_size = config::pipeline_executor_size;
+ if (executors_size <= 0) {
+ executors_size = CpuInfo::num_cores();
+ }
+ auto task_queue = std::make_shared<pipeline::MultiCoreTaskQueue>(executors_size);
+ std::unique_ptr<pipeline::TaskScheduler> pipeline_task_scheduler =
+ std::make_unique<pipeline::TaskScheduler>(
+ exec_env, exec_env->get_global_block_scheduler(), std::move(task_queue),
+ "Pipe_" + tg_name, cg_cpu_ctl_ptr);
+ Status ret = pipeline_task_scheduler->start();
+ if (ret.ok()) {
+ _task_sched = std::move(pipeline_task_scheduler);
+ } else {
+ LOG(INFO) << "[upsert wg thread pool] task scheduler start failed, gid= " << tg_id;
+ }
+ }
+
+ if (_scan_task_sched == nullptr) {
+ std::unique_ptr<vectorized::SimplifiedScanScheduler> scan_scheduler =
+ std::make_unique<vectorized::SimplifiedScanScheduler>("Scan_" + tg_name,
+ cg_cpu_ctl_ptr);
+ Status ret = scan_scheduler->start();
+ if (ret.ok()) {
+ _scan_task_sched = std::move(scan_scheduler);
+ } else {
+ LOG(INFO) << "[upsert wg thread pool] scan scheduler start failed, gid=" << tg_id;
+ }
+ }
+ if (scan_thread_num > 0 && _scan_task_sched) {
+ _scan_task_sched->reset_thread_num(scan_thread_num);
+ }
+
+ if (_remote_scan_task_sched == nullptr) {
+ std::unique_ptr<vectorized::SimplifiedScanScheduler> remote_scan_scheduler =
+ std::make_unique<vectorized::SimplifiedScanScheduler>("RScan_" + tg_name,
+ cg_cpu_ctl_ptr);
+ Status ret = remote_scan_scheduler->start();
+ if (ret.ok()) {
+ _remote_scan_task_sched = std::move(remote_scan_scheduler);
+ } else {
+ LOG(INFO) << "[upsert wg thread pool] remote scan scheduler start failed, gid="
+ << tg_id;
+ }
+ }
+ if (max_remote_scan_thread_num > 0 && _remote_scan_task_sched) {
+ _remote_scan_task_sched->reset_max_thread_num(max_remote_scan_thread_num);
+ }
+ if (min_remote_scan_thread_num > 0 && _remote_scan_task_sched) {
+ _remote_scan_task_sched->reset_min_thread_num(min_remote_scan_thread_num);
+ }
+
+ if (_non_pipe_thread_pool == nullptr) {
+ std::unique_ptr<ThreadPool> thread_pool = nullptr;
+ auto ret = ThreadPoolBuilder("nonPip_" + tg_name)
+ .set_min_threads(1)
+ .set_max_threads(config::fragment_pool_thread_num_max)
+ .set_max_queue_size(config::fragment_pool_queue_size)
+ .set_cgroup_cpu_ctl(cg_cpu_ctl_ptr)
+ .build(&thread_pool);
+ if (!ret.ok()) {
+ LOG(INFO) << "[upsert wg thread pool] create non-pipline thread pool failed, gid="
+ << tg_id;
+ } else {
+ _non_pipe_thread_pool = std::move(thread_pool);
+ }
+ }
+
+ // step 6: update cgroup cpu if needed
+ if (_cgroup_cpu_ctl) {
+ if (enable_cpu_hard_limit) {
+ if (cpu_hard_limit > 0) {
+ _cgroup_cpu_ctl->update_cpu_hard_limit(cpu_hard_limit);
+ _cgroup_cpu_ctl->update_cpu_soft_limit(CPU_SOFT_LIMIT_DEFAULT_VALUE);
+ } else {
+ LOG(INFO) << "[upsert wg thread pool] enable cpu hard limit but value is illegal: "
+ << cpu_hard_limit << ", gid=" << tg_id;
+ }
+ } else {
+ if (config::enable_cgroup_cpu_soft_limit) {
+ _cgroup_cpu_ctl->update_cpu_soft_limit(cpu_shares);
+ _cgroup_cpu_ctl->update_cpu_hard_limit(
+ CPU_HARD_LIMIT_DEFAULT_VALUE); // disable cpu hard limit
+ }
+ }
+ _cgroup_cpu_ctl->get_cgroup_cpu_info(&(tg_info->cgroup_cpu_shares),
+ &(tg_info->cgroup_cpu_hard_limit));
+ }
+}
+
+void TaskGroup::get_query_scheduler(doris::pipeline::TaskScheduler** exec_sched,
+ vectorized::SimplifiedScanScheduler** scan_sched,
+ ThreadPool** non_pipe_thread_pool,
+ vectorized::SimplifiedScanScheduler** remote_scan_sched) {
+ std::shared_lock<std::shared_mutex> rlock(_task_sched_lock);
+ *exec_sched = _task_sched.get();
+ *scan_sched = _scan_task_sched.get();
+ *remote_scan_sched = _remote_scan_task_sched.get();
+ *non_pipe_thread_pool = _non_pipe_thread_pool.get();
+}
+
+void TaskGroup::try_stop_schedulers() {
+ std::shared_lock<std::shared_mutex> rlock(_task_sched_lock);
+ if (_task_sched) {
+ _task_sched->stop();
+ }
+ if (_scan_task_sched) {
+ _scan_task_sched->stop();
+ }
+ if (_remote_scan_task_sched) {
+ _remote_scan_task_sched->stop();
+ }
+ if (_non_pipe_thread_pool) {
+ _non_pipe_thread_pool->shutdown();
+ _non_pipe_thread_pool->wait();
+ }
+}
+
} // namespace taskgroup
} // namespace doris
diff --git a/be/src/runtime/task_group/task_group.h b/be/src/runtime/task_group/task_group.h
index 7604ee45121..c54fce2ab65 100644
--- a/be/src/runtime/task_group/task_group.h
+++ b/be/src/runtime/task_group/task_group.h
@@ -36,9 +36,17 @@ namespace doris {
class MemTrackerLimiter;
class RuntimeProfile;
+class ThreadPool;
+class ExecEnv;
+class CgroupCpuCtl;
+
+namespace vectorized {
+class SimplifiedScanScheduler;
+}
namespace pipeline {
class PipelineTask;
+class TaskScheduler;
} // namespace pipeline
namespace taskgroup {
@@ -121,6 +129,15 @@ public:
int64_t gc_memory(int64_t need_free_mem, RuntimeProfile* profile);
+ void upsert_task_scheduler(taskgroup::TaskGroupInfo* tg_info, ExecEnv* exec_env);
+
+ void get_query_scheduler(doris::pipeline::TaskScheduler** exec_sched,
+ vectorized::SimplifiedScanScheduler** scan_sched,
+ ThreadPool** non_pipe_thread_pool,
+ vectorized::SimplifiedScanScheduler** remote_scan_sched);
+
+ void try_stop_schedulers();
+
private:
mutable std::shared_mutex _mutex; // lock _name, _version, _cpu_share, _memory_limit
const uint64_t _id;
@@ -132,12 +149,21 @@ private:
std::vector<TgTrackerLimiterGroup> _mem_tracker_limiter_pool;
std::atomic<int> _cpu_hard_limit;
std::atomic<int> _scan_thread_num;
+ std::atomic<int> _max_remote_scan_thread_num;
+ std::atomic<int> _min_remote_scan_thread_num;
// means task group is mark dropped
// new query can not submit
// waiting running query to be cancelled or finish
bool _is_shutdown = false;
std::unordered_set<TUniqueId> _query_id_set;
+
+ std::shared_mutex _task_sched_lock;
+ std::unique_ptr<CgroupCpuCtl> _cgroup_cpu_ctl = nullptr;
+ std::unique_ptr<doris::pipeline::TaskScheduler> _task_sched = nullptr;
+ std::unique_ptr<vectorized::SimplifiedScanScheduler> _scan_task_sched = nullptr;
+ std::unique_ptr<vectorized::SimplifiedScanScheduler> _remote_scan_task_sched = nullptr;
+ std::unique_ptr<ThreadPool> _non_pipe_thread_pool = nullptr;
};
using TaskGroupPtr = std::shared_ptr<TaskGroup>;
@@ -152,6 +178,8 @@ struct TaskGroupInfo {
int cpu_hard_limit;
bool enable_cpu_hard_limit;
int scan_thread_num;
+ int max_remote_scan_thread_num;
+ int min_remote_scan_thread_num;
// log cgroup cpu info
uint64_t cgroup_cpu_shares = 0;
int cgroup_cpu_hard_limit = 0;
diff --git a/be/src/runtime/task_group/task_group_manager.cpp b/be/src/runtime/task_group/task_group_manager.cpp
index b0b84a0eb89..819e63c855d 100644
--- a/be/src/runtime/task_group/task_group_manager.cpp
+++ b/be/src/runtime/task_group/task_group_manager.cpp
@@ -21,9 +21,10 @@
#include <mutex>
#include "pipeline/task_scheduler.h"
-#include "runtime/exec_env.h"
#include "runtime/memory/mem_tracker_limiter.h"
#include "runtime/task_group/task_group.h"
+#include "util/threadpool.h"
+#include "util/time.h"
#include "vec/exec/scan/scanner_scheduler.h"
namespace doris::taskgroup {
@@ -68,126 +69,6 @@ TaskGroupPtr TaskGroupManager::get_task_group_by_id(uint64_t tg_id) {
return nullptr;
}
-void TaskGroupManager::get_query_scheduler(uint64_t tg_id,
- doris::pipeline::TaskScheduler** exec_sched,
- vectorized::SimplifiedScanScheduler** scan_sched,
- ThreadPool** non_pipe_thread_pool) {
- std::shared_lock<std::shared_mutex> r_lock(_task_scheduler_lock);
- auto tg_sche_it = _tg_sche_map.find(tg_id);
- if (tg_sche_it != _tg_sche_map.end()) {
- *exec_sched = tg_sche_it->second.get();
- }
-
- auto tg_scan_sche_it = _tg_scan_sche_map.find(tg_id);
- if (tg_scan_sche_it != _tg_scan_sche_map.end()) {
- *scan_sched = tg_scan_sche_it->second.get();
- }
-
- auto non_pipe_thread_pool_iter = _non_pipe_thread_pool_map.find(tg_id);
- if (non_pipe_thread_pool_iter != _non_pipe_thread_pool_map.end()) {
- *non_pipe_thread_pool = non_pipe_thread_pool_iter->second.get();
- }
-}
-
-void TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_info,
- ExecEnv* exec_env) {
- uint64_t tg_id = tg_info->id;
- std::string tg_name = tg_info->name;
- int cpu_hard_limit = tg_info->cpu_hard_limit;
- uint64_t cpu_shares = tg_info->cpu_share;
- bool enable_cpu_hard_limit = tg_info->enable_cpu_hard_limit;
- int scan_thread_num = tg_info->scan_thread_num;
-
- std::lock_guard<std::shared_mutex> write_lock(_task_scheduler_lock);
- // step 1: init cgroup cpu controller
- CgroupCpuCtl* cg_cu_ctl_ptr = nullptr;
- if (config::doris_cgroup_cpu_path != "" &&
- _cgroup_ctl_map.find(tg_id) == _cgroup_ctl_map.end()) {
- std::unique_ptr<CgroupCpuCtl> cgroup_cpu_ctl = std::make_unique<CgroupV1CpuCtl>(tg_id);
- Status ret = cgroup_cpu_ctl->init();
- if (ret.ok()) {
- cg_cu_ctl_ptr = cgroup_cpu_ctl.get();
- _cgroup_ctl_map.emplace(tg_id, std::move(cgroup_cpu_ctl));
- LOG(INFO) << "[upsert wg thread pool] cgroup init success";
- } else {
- LOG(INFO) << "[upsert wg thread pool] cgroup init failed, gid= " << tg_id
- << ", reason=" << ret.to_string();
- }
- }
-
- // step 2: init task scheduler
- if (_tg_sche_map.find(tg_id) == _tg_sche_map.end()) {
- int32_t executors_size = config::pipeline_executor_size;
- if (executors_size <= 0) {
- executors_size = CpuInfo::num_cores();
- }
- auto task_queue = std::make_shared<pipeline::MultiCoreTaskQueue>(executors_size);
-
- auto pipeline_task_scheduler = std::make_unique<pipeline::TaskScheduler>(
- exec_env, exec_env->get_global_block_scheduler(), std::move(task_queue),
- "Exec_" + tg_name, cg_cu_ctl_ptr);
- Status ret = pipeline_task_scheduler->start();
- if (ret.ok()) {
- _tg_sche_map.emplace(tg_id, std::move(pipeline_task_scheduler));
- } else {
- LOG(INFO) << "[upsert wg thread pool] task scheduler start failed, gid= " << tg_id;
- }
- }
-
- // step 3: init scan scheduler
- if (_tg_scan_sche_map.find(tg_id) == _tg_scan_sche_map.end()) {
- auto scan_scheduler =
- std::make_unique<vectorized::SimplifiedScanScheduler>(tg_name, cg_cu_ctl_ptr);
- Status ret = scan_scheduler->start();
- if (ret.ok()) {
- _tg_scan_sche_map.emplace(tg_id, std::move(scan_scheduler));
- } else {
- LOG(INFO) << "[upsert wg thread pool] scan scheduler start failed, gid=" << tg_id;
- }
- }
- if (scan_thread_num > 0 && _tg_scan_sche_map.find(tg_id) != _tg_scan_sche_map.end()) {
- _tg_scan_sche_map.at(tg_id)->reset_thread_num(scan_thread_num);
- }
-
- // step 4: init non-pipe scheduler
- if (_non_pipe_thread_pool_map.find(tg_id) == _non_pipe_thread_pool_map.end()) {
- std::unique_ptr<ThreadPool> thread_pool = nullptr;
- auto ret = ThreadPoolBuilder("nonPip_" + tg_name)
- .set_min_threads(1)
- .set_max_threads(config::fragment_pool_thread_num_max)
- .set_max_queue_size(config::fragment_pool_queue_size)
- .set_cgroup_cpu_ctl(cg_cu_ctl_ptr)
- .build(&thread_pool);
- if (!ret.ok()) {
- LOG(INFO) << "[upsert wg thread pool] create non-pipline thread pool failed, gid="
- << tg_id;
- } else {
- _non_pipe_thread_pool_map.emplace(tg_id, std::move(thread_pool));
- }
- }
-
- // step 5: update cgroup cpu if needed
- if (_cgroup_ctl_map.find(tg_id) != _cgroup_ctl_map.end()) {
- if (enable_cpu_hard_limit) {
- if (cpu_hard_limit > 0) {
- _cgroup_ctl_map.at(tg_id)->update_cpu_hard_limit(cpu_hard_limit);
- _cgroup_ctl_map.at(tg_id)->update_cpu_soft_limit(CPU_SOFT_LIMIT_DEFAULT_VALUE);
- } else {
- LOG(INFO) << "[upsert wg thread pool] enable cpu hard limit but value is illegal: "
- << cpu_hard_limit << ", gid=" << tg_id;
- }
- } else {
- if (config::enable_cgroup_cpu_soft_limit) {
- _cgroup_ctl_map.at(tg_id)->update_cpu_soft_limit(cpu_shares);
- _cgroup_ctl_map.at(tg_id)->update_cpu_hard_limit(
- CPU_HARD_LIMIT_DEFAULT_VALUE); // disable cpu hard limit
- }
- }
- _cgroup_ctl_map.at(tg_id)->get_cgroup_cpu_info(&(tg_info->cgroup_cpu_shares),
- &(tg_info->cgroup_cpu_hard_limit));
- }
-}
-
void TaskGroupManager::delete_task_group_by_ids(std::set<uint64_t> used_wg_id) {
int64_t begin_time = MonotonicMillis();
// 1 get delete group without running queries
@@ -209,45 +90,11 @@ void TaskGroupManager::delete_task_group_by_ids(std::set<uint64_t> used_wg_id) {
}
// 2 stop active thread
- std::vector<doris::pipeline::TaskScheduler*> task_sched_to_stop;
- std::vector<vectorized::SimplifiedScanScheduler*> scan_task_sched_to_stop;
- std::vector<ThreadPool*> non_pip_thread_pool_to_stop;
- {
- std::shared_lock<std::shared_mutex> read_lock(_task_scheduler_lock);
- for (uint64_t tg_id : deleted_tg_ids) {
- if (_tg_sche_map.find(tg_id) != _tg_sche_map.end()) {
- task_sched_to_stop.emplace_back(_tg_sche_map.at(tg_id).get());
- }
- if (_tg_scan_sche_map.find(tg_id) != _tg_scan_sche_map.end()) {
- scan_task_sched_to_stop.emplace_back(_tg_scan_sche_map.at(tg_id).get());
- }
- if (_non_pipe_thread_pool_map.find(tg_id) != _non_pipe_thread_pool_map.end()) {
- non_pip_thread_pool_to_stop.emplace_back(_non_pipe_thread_pool_map.at(tg_id).get());
- }
- }
- }
- for (auto* ptr1 : task_sched_to_stop) {
- ptr1->stop();
- }
- for (auto* ptr2 : scan_task_sched_to_stop) {
- ptr2->stop();
- }
- for (auto& ptr3 : non_pip_thread_pool_to_stop) {
- ptr3->shutdown();
- ptr3->wait();
+ for (uint64_t tg_id : deleted_tg_ids) {
+ _task_groups.at(tg_id)->try_stop_schedulers();
}
// 3 release resource in memory
- {
- std::lock_guard<std::shared_mutex> write_lock(_task_scheduler_lock);
- for (uint64_t tg_id : deleted_tg_ids) {
- _tg_sche_map.erase(tg_id);
- _tg_scan_sche_map.erase(tg_id);
- _cgroup_ctl_map.erase(tg_id);
- _non_pipe_thread_pool_map.erase(tg_id);
- }
- }
-
{
std::lock_guard<std::shared_mutex> write_lock(_group_mutex);
for (uint64_t tg_id : deleted_tg_ids) {
@@ -286,14 +133,8 @@ void TaskGroupManager::delete_task_group_by_ids(std::set<uint64_t> used_wg_id) {
}
void TaskGroupManager::stop() {
- for (auto& task_sche : _tg_sche_map) {
- task_sche.second->stop();
- }
- for (auto& task_sche : _tg_scan_sche_map) {
- task_sche.second->stop();
- }
- for (auto& no_pip_sche : _non_pipe_thread_pool_map) {
- no_pip_sche.second->shutdown();
+ for (auto iter = _task_groups.begin(); iter != _task_groups.end(); iter++) {
+ iter->second->try_stop_schedulers();
}
}
diff --git a/be/src/runtime/task_group/task_group_manager.h b/be/src/runtime/task_group/task_group_manager.h
index 29e5c30a596..21772bd3bca 100644
--- a/be/src/runtime/task_group/task_group_manager.h
+++ b/be/src/runtime/task_group/task_group_manager.h
@@ -21,18 +21,11 @@
#include <shared_mutex>
#include <unordered_map>
-#include "pipeline/task_queue.h"
-#include "pipeline/task_scheduler.h"
#include "task_group.h"
namespace doris {
-class ExecEnv;
-class QueryContext;
-class CgroupCpuCtl;
-namespace vectorized {
-class SimplifiedScanScheduler;
-}
+class CgroupCpuCtl;
namespace pipeline {
class TaskScheduler;
@@ -51,8 +44,6 @@ public:
void get_related_taskgroups(const std::function<bool(const TaskGroupPtr& ptr)>& pred,
std::vector<TaskGroupPtr>* task_groups);
- void upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_info, ExecEnv* exec_env);
-
void delete_task_group_by_ids(std::set<uint64_t> id_set);
TaskGroupPtr get_task_group_by_id(uint64_t tg_id);
@@ -65,22 +56,10 @@ public:
bool enable_cpu_hard_limit() { return _enable_cpu_hard_limit.load(); }
- void get_query_scheduler(uint64_t tg_id, doris::pipeline::TaskScheduler** exec_sched,
- vectorized::SimplifiedScanScheduler** scan_sched,
- ThreadPool** non_pipe_thread_pool);
-
private:
std::shared_mutex _group_mutex;
std::unordered_map<uint64_t, TaskGroupPtr> _task_groups;
- // map for workload group id and task scheduler pool
- // used for cpu hard limit
- std::shared_mutex _task_scheduler_lock;
- std::map<uint64_t, std::unique_ptr<doris::pipeline::TaskScheduler>> _tg_sche_map;
- std::map<uint64_t, std::unique_ptr<vectorized::SimplifiedScanScheduler>> _tg_scan_sche_map;
- std::map<uint64_t, std::unique_ptr<CgroupCpuCtl>> _cgroup_ctl_map;
- std::map<uint64_t, std::unique_ptr<ThreadPool>> _non_pipe_thread_pool_map;
-
std::shared_mutex _init_cg_ctl_lock;
std::unique_ptr<CgroupCpuCtl> _cg_cpu_ctl;
bool _is_init_succ = false;
diff --git a/be/src/vec/exec/scan/scanner_context.cpp b/be/src/vec/exec/scan/scanner_context.cpp
index 45e934ee790..be700abc7bf 100644
--- a/be/src/vec/exec/scan/scanner_context.cpp
+++ b/be/src/vec/exec/scan/scanner_context.cpp
@@ -128,6 +128,7 @@ Status ScannerContext::init() {
if (_simple_scan_scheduler) {
_should_reset_thread_name = false;
}
+ _remote_scan_task_scheduler = _state->get_query_ctx()->get_remote_scan_scheduler();
}
#endif
diff --git a/be/src/vec/exec/scan/scanner_context.h b/be/src/vec/exec/scan/scanner_context.h
index ca80d6e748c..7b021013bbb 100644
--- a/be/src/vec/exec/scan/scanner_context.h
+++ b/be/src/vec/exec/scan/scanner_context.h
@@ -157,6 +157,8 @@ public:
SimplifiedScanScheduler* get_simple_scan_scheduler() { return _simple_scan_scheduler; }
+ SimplifiedScanScheduler* get_remote_scan_scheduler() { return _remote_scan_task_scheduler; }
+
void stop_scanners(RuntimeState* state);
int32_t get_max_thread_num() const { return _max_thread_num; }
@@ -215,6 +217,7 @@ protected:
int64_t _max_bytes_in_queue;
doris::vectorized::ScannerScheduler* _scanner_scheduler;
SimplifiedScanScheduler* _simple_scan_scheduler = nullptr;
+ SimplifiedScanScheduler* _remote_scan_task_scheduler = nullptr;
moodycamel::ConcurrentQueue<std::weak_ptr<ScannerDelegate>> _scanners;
int32_t _num_scheduled_scanners = 0;
int32_t _num_finished_scanners = 0;
diff --git a/be/src/vec/exec/scan/scanner_scheduler.cpp b/be/src/vec/exec/scan/scanner_scheduler.cpp
index d3fd740b437..00c900fc978 100644
--- a/be/src/vec/exec/scan/scanner_scheduler.cpp
+++ b/be/src/vec/exec/scan/scanner_scheduler.cpp
@@ -157,12 +157,12 @@ void ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx,
TabletStorageType type = scanner_delegate->_scanner->get_storage_type();
bool ret = false;
if (type == TabletStorageType::STORAGE_TYPE_LOCAL) {
- if (auto* scan_sche = ctx->get_simple_scan_scheduler()) {
+ if (auto* scan_sched = ctx->get_simple_scan_scheduler()) {
auto work_func = [this, scanner_ref = scan_task, ctx]() {
this->_scanner_scan(ctx, scanner_ref);
};
SimplifiedScanTask simple_scan_task = {work_func, ctx};
- ret = scan_sche->submit_scan_task(simple_scan_task);
+ ret = scan_sched->submit_scan_task(simple_scan_task);
} else {
PriorityThreadPool::Task task;
task.work_function = [this, scanner_ref = scan_task, ctx]() {
@@ -172,12 +172,20 @@ void ScannerScheduler::submit(std::shared_ptr<ScannerContext> ctx,
ret = _local_scan_thread_pool->offer(task);
}
} else {
- PriorityThreadPool::Task task;
- task.work_function = [this, scanner_ref = scan_task, ctx]() {
- this->_scanner_scan(ctx, scanner_ref);
- };
- task.priority = nice;
- ret = _remote_scan_thread_pool->offer(task);
+ if (auto* remote_scan_sched = ctx->get_remote_scan_scheduler()) {
+ auto work_func = [this, scanner_ref = scan_task, ctx]() {
+ this->_scanner_scan(ctx, scanner_ref);
+ };
+ SimplifiedScanTask simple_scan_task = {work_func, ctx};
+ ret = remote_scan_sched->submit_scan_task(simple_scan_task);
+ } else {
+ PriorityThreadPool::Task task;
+ task.work_function = [this, scanner_ref = scan_task, ctx]() {
+ this->_scanner_scan(ctx, scanner_ref);
+ };
+ task.priority = nice;
+ ret = _remote_scan_thread_pool->offer(task);
+ }
}
if (!ret) {
scan_task->set_status(
diff --git a/be/src/vec/exec/scan/scanner_scheduler.h b/be/src/vec/exec/scan/scanner_scheduler.h
index 746aa34ff9a..f3f9caaa4d3 100644
--- a/be/src/vec/exec/scan/scanner_scheduler.h
+++ b/be/src/vec/exec/scan/scanner_scheduler.h
@@ -101,15 +101,15 @@ struct SimplifiedScanTask {
class SimplifiedScanScheduler {
public:
- SimplifiedScanScheduler(std::string wg_name, CgroupCpuCtl* cgroup_cpu_ctl) {
+ SimplifiedScanScheduler(std::string sched_name, CgroupCpuCtl* cgroup_cpu_ctl) {
_is_stop.store(false);
_cgroup_cpu_ctl = cgroup_cpu_ctl;
- _wg_name = wg_name;
+ _sched_name = sched_name;
}
~SimplifiedScanScheduler() {
stop();
- LOG(INFO) << "Scanner sche " << _wg_name << " shutdown";
+ LOG(INFO) << "Scanner sche " << _sched_name << " shutdown";
}
void stop() {
@@ -119,7 +119,7 @@ public:
}
Status start() {
- RETURN_IF_ERROR(ThreadPoolBuilder("Scan_" + _wg_name)
+ RETURN_IF_ERROR(ThreadPoolBuilder(_sched_name)
.set_min_threads(config::doris_scanner_thread_pool_thread_num)
.set_max_threads(config::doris_scanner_thread_pool_thread_num)
.set_cgroup_cpu_ctl(_cgroup_cpu_ctl)
@@ -131,7 +131,7 @@ public:
if (!_is_stop) {
return _scan_thread_pool->submit_func([scan_task] { scan_task.scan_func(); });
} else {
- return Status::InternalError<false>("scanner pool {} is shutdown.", _wg_name);
+ return Status::InternalError<false>("scanner pool {} is shutdown.", _sched_name);
}
}
@@ -148,11 +148,33 @@ public:
}
}
+ void reset_max_thread_num(int thread_num) {
+ int max_thread_num = _scan_thread_pool->max_threads();
+
+ if (max_thread_num != thread_num) {
+ Status st = _scan_thread_pool->set_max_threads(thread_num);
+ if (!st.ok()) {
+ LOG(INFO) << "reset max thread num failed, sche name=" << _sched_name;
+ }
+ }
+ }
+
+ void reset_min_thread_num(int thread_num) {
+ int min_thread_num = _scan_thread_pool->min_threads();
+
+ if (min_thread_num != thread_num) {
+ Status st = _scan_thread_pool->set_min_threads(thread_num);
+ if (!st.ok()) {
+ LOG(INFO) << "reset min thread num failed, sche name=" << _sched_name;
+ }
+ }
+ }
+
private:
std::unique_ptr<ThreadPool> _scan_thread_pool;
std::atomic<bool> _is_stop;
CgroupCpuCtl* _cgroup_cpu_ctl = nullptr;
- std::string _wg_name;
+ std::string _sched_name;
};
} // namespace doris::vectorized
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java
index 0def933c8cf..b14b3afec03 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java
@@ -63,12 +63,17 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
public static final String SCAN_THREAD_NUM = "scan_thread_num";
+ public static final String MAX_REMOTE_SCAN_THREAD_NUM = "max_remote_scan_thread_num";
+
+ public static final String MIN_REMOTE_SCAN_THREAD_NUM = "min_remote_scan_thread_num";
+
// NOTE(wb): all property is not required, some properties default value is set in be
// default value is as followed
// cpu_share=1024, memory_limit=0%(0 means not limit), enable_memory_overcommit=true
private static final ImmutableSet<String> ALL_PROPERTIES_NAME = new ImmutableSet.Builder<String>()
.add(CPU_SHARE).add(MEMORY_LIMIT).add(ENABLE_MEMORY_OVERCOMMIT).add(MAX_CONCURRENCY)
- .add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM).build();
+ .add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM)
+ .add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM).build();
@SerializedName(value = "id")
private long id;
@@ -225,6 +230,32 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
}
}
+ if (properties.containsKey(MAX_REMOTE_SCAN_THREAD_NUM)) {
+ String value = properties.get(MAX_REMOTE_SCAN_THREAD_NUM);
+ try {
+ int intValue = Integer.parseInt(value);
+ if (intValue <= 0 && intValue != -1) {
+ throw new NumberFormatException();
+ }
+ } catch (NumberFormatException e) {
+ throw new DdlException(
+ MAX_REMOTE_SCAN_THREAD_NUM + " must be a positive integer or -1. but input value is " + value);
+ }
+ }
+
+ if (properties.containsKey(MIN_REMOTE_SCAN_THREAD_NUM)) {
+ String value = properties.get(MIN_REMOTE_SCAN_THREAD_NUM);
+ try {
+ int intValue = Integer.parseInt(value);
+ if (intValue <= 0 && intValue != -1) {
+ throw new NumberFormatException();
+ }
+ } catch (NumberFormatException e) {
+ throw new DdlException(
+ MAX_REMOTE_SCAN_THREAD_NUM + " must be a positive integer or -1. but input value is " + value);
+ }
+ }
+
// check queue property
if (properties.containsKey(MAX_CONCURRENCY)) {
try {
@@ -309,6 +340,10 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
row.add("true");
} else if (SCAN_THREAD_NUM.equals(key) && !properties.containsKey(key)) {
row.add("-1");
+ } else if (MAX_REMOTE_SCAN_THREAD_NUM.equals(key) && !properties.containsKey(key)) {
+ row.add("-1");
+ } else if (MIN_REMOTE_SCAN_THREAD_NUM.equals(key) && !properties.containsKey(key)) {
+ row.add("-1");
} else {
row.add(properties.get(key));
}
@@ -369,6 +404,16 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
tWorkloadGroupInfo.setScanThreadNum(Integer.parseInt(scanThreadNumStr));
}
+ String maxRemoteScanThreadNumStr = properties.get(MAX_REMOTE_SCAN_THREAD_NUM);
+ if (maxRemoteScanThreadNumStr != null) {
+ tWorkloadGroupInfo.setMaxRemoteScanThreadNum(Integer.parseInt(maxRemoteScanThreadNumStr));
+ }
+
+ String minRemoteScanThreadNumStr = properties.get(MIN_REMOTE_SCAN_THREAD_NUM);
+ if (minRemoteScanThreadNumStr != null) {
+ tWorkloadGroupInfo.setMinRemoteScanThreadNum(Integer.parseInt(minRemoteScanThreadNumStr));
+ }
+
TopicInfo topicInfo = new TopicInfo();
topicInfo.setWorkloadGroupInfo(tWorkloadGroupInfo);
return topicInfo;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
index 60d7e296ed3..a7c26f7cec5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
@@ -70,7 +70,8 @@ public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
.add(WorkloadGroup.ENABLE_MEMORY_OVERCOMMIT)
.add(WorkloadGroup.MAX_CONCURRENCY).add(WorkloadGroup.MAX_QUEUE_SIZE)
.add(WorkloadGroup.QUEUE_TIMEOUT).add(WorkloadGroup.CPU_HARD_LIMIT)
- .add(WorkloadGroup.SCAN_THREAD_NUM)
+ .add(WorkloadGroup.SCAN_THREAD_NUM).add(WorkloadGroup.MAX_REMOTE_SCAN_THREAD_NUM)
+ .add(WorkloadGroup.MIN_REMOTE_SCAN_THREAD_NUM)
.add(QueryQueue.RUNNING_QUERY_NUM).add(QueryQueue.WAITING_QUERY_NUM)
.build();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
index 6feb8fa94f4..8a3df743e24 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
@@ -378,14 +378,18 @@ public class MetadataGenerator {
trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(1))); // name
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(2)))); // cpu_share
trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(3))); // mem_limit
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(4))); //mem overcommit
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(4))); // mem overcommit
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(5)))); // max concurrent
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(6)))); // max queue size
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(7)))); // queue timeout
trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(8))); // cpu hard limit
trow.addToColumnValue(new TCell().setIntVal(Integer.parseInt(rGroupsInfo.get(9)))); // scan thread num
- trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(10)))); // running query num
- trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(11)))); // waiting query num
+ // max remote scan thread num
+ trow.addToColumnValue(new TCell().setIntVal(Integer.parseInt(rGroupsInfo.get(10))));
+ // min remote scan thread num
+ trow.addToColumnValue(new TCell().setIntVal(Integer.parseInt(rGroupsInfo.get(11))));
+ trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(12)))); // running query num
+ trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(13)))); // waiting query num
dataBatch.add(trow);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/WorkloadGroupsTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/WorkloadGroupsTableValuedFunction.java
index a50fb7ca853..27e011d60d5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/WorkloadGroupsTableValuedFunction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/WorkloadGroupsTableValuedFunction.java
@@ -50,6 +50,8 @@ public class WorkloadGroupsTableValuedFunction extends MetadataTableValuedFuncti
new Column(WorkloadGroup.QUEUE_TIMEOUT, ScalarType.createType(PrimitiveType.BIGINT)),
new Column(WorkloadGroup.CPU_HARD_LIMIT, ScalarType.createStringType()),
new Column(WorkloadGroup.SCAN_THREAD_NUM, ScalarType.createType(PrimitiveType.INT)),
+ new Column(WorkloadGroup.MAX_REMOTE_SCAN_THREAD_NUM, ScalarType.createType(PrimitiveType.INT)),
+ new Column(WorkloadGroup.MIN_REMOTE_SCAN_THREAD_NUM, ScalarType.createType(PrimitiveType.INT)),
new Column(QueryQueue.RUNNING_QUERY_NUM, ScalarType.createType(PrimitiveType.BIGINT)),
new Column(QueryQueue.WAITING_QUERY_NUM, ScalarType.createType(PrimitiveType.BIGINT)));
diff --git a/gensrc/thrift/BackendService.thrift b/gensrc/thrift/BackendService.thrift
index 24edaefc103..b803618af49 100644
--- a/gensrc/thrift/BackendService.thrift
+++ b/gensrc/thrift/BackendService.thrift
@@ -177,6 +177,8 @@ struct TWorkloadGroupInfo {
7: optional bool enable_memory_overcommit
8: optional bool enable_cpu_hard_limit
9: optional i32 scan_thread_num
+ 10: optional i32 max_remote_scan_thread_num
+ 11: optional i32 min_remote_scan_thread_num
}
enum TWorkloadMetricType {
diff --git a/gensrc/thrift/FrontendService.thrift b/gensrc/thrift/FrontendService.thrift
index 88cf25b9c67..1e433d42f5d 100644
--- a/gensrc/thrift/FrontendService.thrift
+++ b/gensrc/thrift/FrontendService.thrift
@@ -412,7 +412,7 @@ struct TQueryStatistics {
struct TReportWorkloadRuntimeStatusParams {
1: optional i64 backend_id
- 2: map<string, TQueryStatistics> query_statistics_map
+ 2: optional map<string, TQueryStatistics> query_statistics_map
}
// The results of an INSERT query, sent to the coordinator as part of
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/06: [fix](function) fix unexpected be core in string search function (#31312)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b8dafb20017e2a078448a16b1c9b9039323fb3f6
Author: xy720 <22...@users.noreply.github.com>
AuthorDate: Mon Feb 26 16:33:59 2024 +0800
[fix](function) fix unexpected be core in string search function (#31312)
Fix be core in multi_match_any/multi_search_all_positions functions.
---
.../functions/functions_multi_string_position.cpp | 15 +++++++------
.../functions/functions_multi_string_search.cpp | 15 +++++++------
.../apache/doris/analysis/FunctionCallExpr.java | 3 +++
.../java/org/apache/doris/catalog/FunctionSet.java | 6 +++++
.../test_multi_string_position.groovy | 24 ++++++++++++++++++++
.../test_multi_string_search.groovy | 26 +++++++++++++++++++++-
6 files changed, 74 insertions(+), 15 deletions(-)
diff --git a/be/src/vec/functions/functions_multi_string_position.cpp b/be/src/vec/functions/functions_multi_string_position.cpp
index 904fbb2e236..4498147c67b 100644
--- a/be/src/vec/functions/functions_multi_string_position.cpp
+++ b/be/src/vec/functions/functions_multi_string_position.cpp
@@ -80,26 +80,22 @@ public:
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) const override {
auto haystack_column = block.get_by_position(arguments[0]).column;
- auto haystack_ptr = haystack_column;
-
auto needles_column = block.get_by_position(arguments[1]).column;
- auto needles_ptr = needles_column;
bool haystack_nullable = false;
bool needles_nullable = false;
if (haystack_column->is_nullable()) {
- haystack_ptr = check_and_get_column<ColumnNullable>(haystack_column.get())
- ->get_nested_column_ptr();
haystack_nullable = true;
}
if (needles_column->is_nullable()) {
- needles_ptr = check_and_get_column<ColumnNullable>(needles_column.get())
- ->get_nested_column_ptr();
needles_nullable = true;
}
+ auto haystack_ptr = remove_nullable(haystack_column);
+ auto needles_ptr = remove_nullable(needles_column);
+
const ColumnString* col_haystack_vector =
check_and_get_column<ColumnString>(&*haystack_ptr);
const ColumnConst* col_haystack_const =
@@ -110,6 +106,11 @@ public:
const ColumnConst* col_needles_const =
check_and_get_column_const<ColumnArray>(needles_ptr.get());
+ if (!col_needles_const && !col_needles_vector)
+ return Status::InvalidArgument(
+ "function '{}' encountered unsupported needles column, found {}", name,
+ needles_column->get_name());
+
if (col_haystack_const && col_needles_vector) {
return Status::InvalidArgument(
"function '{}' doesn't support search with non-constant needles "
diff --git a/be/src/vec/functions/functions_multi_string_search.cpp b/be/src/vec/functions/functions_multi_string_search.cpp
index 72c0eeed87e..f7a1b8d7a90 100644
--- a/be/src/vec/functions/functions_multi_string_search.cpp
+++ b/be/src/vec/functions/functions_multi_string_search.cpp
@@ -78,26 +78,22 @@ public:
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
size_t result, size_t input_rows_count) const override {
auto haystack_column = block.get_by_position(arguments[0]).column;
- auto haystack_ptr = haystack_column;
-
auto needles_column = block.get_by_position(arguments[1]).column;
- auto needles_ptr = needles_column;
bool haystack_nullable = false;
bool needles_nullable = false;
if (haystack_column->is_nullable()) {
- haystack_ptr = check_and_get_column<ColumnNullable>(haystack_column.get())
- ->get_nested_column_ptr();
haystack_nullable = true;
}
if (needles_column->is_nullable()) {
- needles_ptr = check_and_get_column<ColumnNullable>(needles_column.get())
- ->get_nested_column_ptr();
needles_nullable = true;
}
+ auto haystack_ptr = remove_nullable(haystack_column);
+ auto needles_ptr = remove_nullable(needles_column);
+
const ColumnString* col_haystack_vector =
check_and_get_column<ColumnString>(&*haystack_ptr);
const ColumnConst* col_haystack_const =
@@ -108,6 +104,11 @@ public:
const ColumnConst* col_needles_const =
check_and_get_column_const<ColumnArray>(needles_ptr.get());
+ if (!col_needles_const && !col_needles_vector)
+ return Status::InvalidArgument(
+ "function '{}' encountered unsupported needles column, found {}", name,
+ needles_column->get_name());
+
if (col_haystack_const && col_needles_vector)
return Status::InvalidArgument(
"function '{}' doesn't support search with non-constant needles "
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index 9907b497bb9..1460adaf048 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -84,6 +84,9 @@ public class FunctionCallExpr extends Expr {
String.CASE_INSENSITIVE_ORDER)
.add("round").add("round_bankers").add("ceil").add("floor")
.add("truncate").add("dround").add("dceil").add("dfloor").build();
+ public static final ImmutableSet<String> STRING_SEARCH_FUNCTION_SET = new ImmutableSortedSet.Builder(
+ String.CASE_INSENSITIVE_ORDER)
+ .add("multi_search_all_positions").add("multi_match_any").build();
private final AtomicBoolean addOnce = new AtomicBoolean(false);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
index babd697680c..b69d52b2c11 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
@@ -495,6 +495,12 @@ public class FunctionSet<T> {
return false;
}
}
+ if (FunctionCallExpr.STRING_SEARCH_FUNCTION_SET.contains(desc.functionName())) {
+ if (descArgTypes[1].isStringType() && candicateArgTypes[1].isArrayType()) {
+ // The needles arg of search functions should not be allowed to cast from string.
+ return false;
+ }
+ }
// If set `roundPreciseDecimalV2Value`, only use decimalv3 as target type to execute round function
if (ConnectContext.get() != null
&& ConnectContext.get().getSessionVariable().roundPreciseDecimalV2Value
diff --git a/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_position.groovy b/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_position.groovy
index f6a8aa110e9..1c82fc7c6e9 100644
--- a/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_position.groovy
+++ b/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_position.groovy
@@ -54,4 +54,28 @@ suite("test_multi_string_position") {
qt_select3 "select multi_search_all_positions('mpswgtljbbrmivkcglamemayfn', ['', 'm', 'saejhpnfgfq', 'rzanrkdssmmkanqjpfi', 'oputeneprgoowg', 'mp', '', '', 'wgtljbbrmivkcglamemay', 'cbpthtrgrmgfypizi', 'tl', 'tlj', 'xuhs', 'brmivkcglamemayfn', '', 'gtljb'])"
qt_select4 "select multi_search_all_positions('arbphzbbecypbzsqsljurtddve', ['arbphzb', 'mnrboimjfijnti', 'cikcrd', 'becypbz', 'z', 'uocmqgnczhdcrvtqrnaxdxjjlhakoszuwc', 'bbe', '', 'bp', 'yhltnexlpdijkdzt', 'jkwjmrckvgmccmmrolqvy', 'vdxmicjmfbtsbqqmqcgtnrvdgaucsgspwg', 'witlfqwvhmmyjrnrzttrikhhsrd', 'pbzsqsljurt'])"
qt_select5 "select multi_search_all_positions('aizovxqpzcbbxuhwtiaaqhdqjdei', ['qpzcbbxuhw', 'jugrpglqbm', 'dspwhzpyjohhtizegrnswhjfpdz', 'pzcbbxuh', 'vayzeszlycke', 'i', 'gvrontcpqavsjxtjwzgwxugiyhkhmhq', 'gyzmeroxztgaurmrqwtmsxcqnxaezuoapatvu', 'xqpzc', 'mjiswsvlvlpqrhhptqq', 'iz', 'hmzjxxfjsvcvdpqwtrdrp', 'zovxqpzcbbxuhwtia', 'ai'])"
+
+ try {
+ sql "select multi_search_all_positions(content, 'hello') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_search_all_positions"))
+ }
+
+ try {
+ sql "select multi_search_all_positions(content, 'hello, !, world, Hello, World') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_search_all_positions"))
+ }
+
+ try {
+ sql "select multi_search_all_positions(content, '[hello]') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_search_all_positions"))
+ }
+
+ try {
+ sql "select multi_search_all_positions(content, '[hello, !, world, Hello, World]') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_search_all_positions"))
+ }
}
diff --git a/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_search.groovy b/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_search.groovy
index 111fcaf7173..55a9f6894fd 100644
--- a/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_search.groovy
+++ b/regression-test/suites/query_p0/sql_functions/search_functions/test_multi_string_search.groovy
@@ -71,4 +71,28 @@ suite("test_multi_string_search", "arrow_flight_sql") {
qt_select "select multi_match_any('ldrzgttlqaphekkkdukgngl', ['gttlqaphekkkdukgn', 'ekkkd', 'gttlqaphe', 'qaphek', 'h', 'kdu', 'he', 'phek', '', 'drzgttlqaphekkkd'])"
qt_select "select multi_match_any('ololo', ['ololo', 'ololo', 'ololo'])"
qt_select "select multi_match_any('khljxzxlpcrxpkrfybbfk', ['k'])"
-}
\ No newline at end of file
+
+ try {
+ sql "select multi_match_any(content, 'hello') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_match_any"))
+ }
+
+ try {
+ sql "select multi_match_any(content, 'hello, !, world, Hello, World') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_match_any"))
+ }
+
+ try {
+ sql "select multi_match_any(content, '[hello]') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_match_any"))
+ }
+
+ try {
+ sql "select multi_match_any(content, '[hello, !, world, Hello, World]') from ${table_name} order by col1"
+ } catch (Exception ex) {
+ assert("${ex}".contains("errCode = 2, detailMessage = No matching function with signature: multi_match_any"))
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/06: [Improment](executor)Add scanbytes/scanrows condition (#31364)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2a10c5d2f65e5f244d2c09e6eb9f0f8f4271a8da
Author: wangbo <wa...@apache.org>
AuthorDate: Mon Feb 26 19:01:35 2024 +0800
[Improment](executor)Add scanbytes/scanrows condition (#31364)
* Add scanbytes/scanrows condition
* fix reg
---
be/src/agent/workload_sched_policy_listener.cpp | 4 +-
be/src/runtime/fragment_mgr.cpp | 7 --
be/src/runtime/runtime_query_statistics_mgr.cpp | 19 ++++
be/src/runtime/runtime_query_statistics_mgr.h | 8 ++
.../runtime/workload_management/workload_action.h | 4 +-
.../workload_management/workload_condition.h | 4 +-
.../workload_sched_policy_mgr.cpp | 5 +
.../main/java/org/apache/doris/common/Config.java | 3 -
.../main/java/org/apache/doris/catalog/Env.java | 6 -
.../publish/WorkloadActionPublishThread.java | 123 ---------------------
.../workloadschedpolicy/WorkloadAction.java | 2 -
.../WorkloadActionMoveQueryToGroup.java | 67 -----------
.../workloadschedpolicy/WorkloadCondition.java | 4 +
...tion.java => WorkloadConditionBeScanBytes.java} | 32 ++++--
...ition.java => WorkloadConditionBeScanRows.java} | 32 ++++--
.../workloadschedpolicy/WorkloadConditionMeta.java | 4 +
.../workloadschedpolicy/WorkloadMetricType.java | 2 +-
.../workloadschedpolicy/WorkloadSchedPolicy.java | 4 +-
.../WorkloadSchedPolicyMgr.java | 22 +---
gensrc/thrift/BackendService.thrift | 12 +-
.../test_workload_sched_policy.out | 1 -
.../test_workload_sched_policy.groovy | 21 +---
22 files changed, 101 insertions(+), 285 deletions(-)
diff --git a/be/src/agent/workload_sched_policy_listener.cpp b/be/src/agent/workload_sched_policy_listener.cpp
index 461fd2cbb0f..689a600367a 100644
--- a/be/src/agent/workload_sched_policy_listener.cpp
+++ b/be/src/agent/workload_sched_policy_listener.cpp
@@ -69,9 +69,7 @@ void WorkloadschedPolicyListener::handle_topic_info(const std::vector<TopicInfo>
policy_map.emplace(tpolicy.id, std::move(policy_ptr));
}
size_t new_policy_size = policy_map.size();
- if (new_policy_size > 0) {
- _exec_env->workload_sched_policy_mgr()->update_workload_sched_policy(std::move(policy_map));
- }
+ _exec_env->workload_sched_policy_mgr()->update_workload_sched_policy(std::move(policy_map));
LOG(INFO) << "[workload_schedule]finish update workload schedule policy, size="
<< new_policy_size;
}
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index f6985538854..e52b71d277b 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -1506,17 +1506,10 @@ void FragmentMgr::_setup_shared_hashtable_for_broadcast_join(const TPipelineFrag
void FragmentMgr::get_runtime_query_info(std::vector<WorkloadQueryInfo>* query_info_list) {
{
std::lock_guard<std::mutex> lock(_lock);
- // todo: use monotonic time
- VecDateTimeValue now = VecDateTimeValue::local_time();
for (const auto& q : _query_ctx_map) {
WorkloadQueryInfo workload_query_info;
workload_query_info.query_id = print_id(q.first);
workload_query_info.tquery_id = q.first;
-
- uint64_t query_time_millisecond = q.second->query_time(now) * 1000;
- workload_query_info.metric_map.emplace(WorkloadMetricType::QUERY_TIME,
- std::to_string(query_time_millisecond));
- // todo, add scan rows, scan bytes
query_info_list->push_back(workload_query_info);
}
}
diff --git a/be/src/runtime/runtime_query_statistics_mgr.cpp b/be/src/runtime/runtime_query_statistics_mgr.cpp
index a658e527f61..c6f70643f13 100644
--- a/be/src/runtime/runtime_query_statistics_mgr.cpp
+++ b/be/src/runtime/runtime_query_statistics_mgr.cpp
@@ -172,6 +172,25 @@ std::shared_ptr<QueryStatistics> RuntimeQueryStatiticsMgr::get_runtime_query_sta
return qs_ptr;
}
+void RuntimeQueryStatiticsMgr::get_metric_map(
+ std::string query_id, std::map<WorkloadMetricType, std::string>& metric_map) {
+ QueryStatistics ret_qs;
+ int64_t query_time_ms = 0;
+ {
+ std::shared_lock<std::shared_mutex> read_lock(_qs_ctx_map_lock);
+ if (_query_statistics_ctx_map.find(query_id) != _query_statistics_ctx_map.end()) {
+ for (auto const& qs : _query_statistics_ctx_map[query_id]->_qs_list) {
+ ret_qs.merge(*qs);
+ }
+ query_time_ms =
+ MonotonicMillis() - _query_statistics_ctx_map.at(query_id)->_query_start_time;
+ }
+ }
+ metric_map.emplace(WorkloadMetricType::QUERY_TIME, std::to_string(query_time_ms));
+ metric_map.emplace(WorkloadMetricType::SCAN_ROWS, std::to_string(ret_qs.get_scan_rows()));
+ metric_map.emplace(WorkloadMetricType::SCAN_BYTES, std::to_string(ret_qs.get_scan_bytes()));
+}
+
void RuntimeQueryStatiticsMgr::set_workload_group_id(std::string query_id, int64_t wg_id) {
// wg id just need eventual consistency, read lock is ok
std::shared_lock<std::shared_mutex> read_lock(_qs_ctx_map_lock);
diff --git a/be/src/runtime/runtime_query_statistics_mgr.h b/be/src/runtime/runtime_query_statistics_mgr.h
index 98d4f554728..69b283b6d14 100644
--- a/be/src/runtime/runtime_query_statistics_mgr.h
+++ b/be/src/runtime/runtime_query_statistics_mgr.h
@@ -21,6 +21,8 @@
#include <string>
#include "runtime/query_statistics.h"
+#include "runtime/workload_management/workload_condition.h"
+#include "util/time.h"
namespace doris {
@@ -29,6 +31,7 @@ public:
QueryStatisticsCtx(TNetworkAddress fe_addr) : _fe_addr(fe_addr) {
this->_is_query_finished = false;
this->_wg_id = -1;
+ this->_query_start_time = MonotonicMillis();
}
~QueryStatisticsCtx() = default;
@@ -40,6 +43,7 @@ public:
TNetworkAddress _fe_addr;
int64_t _query_finish_time;
int64_t _wg_id;
+ int64_t _query_start_time;
};
class RuntimeQueryStatiticsMgr {
@@ -58,6 +62,10 @@ public:
void set_workload_group_id(std::string query_id, int64_t wg_id);
+ // used for workload scheduler policy
+ void get_metric_map(std::string query_id,
+ std::map<WorkloadMetricType, std::string>& metric_map);
+
private:
std::shared_mutex _qs_ctx_map_lock;
std::map<std::string, std::unique_ptr<QueryStatisticsCtx>> _query_statistics_ctx_map;
diff --git a/be/src/runtime/workload_management/workload_action.h b/be/src/runtime/workload_management/workload_action.h
index 29c01320b78..785acc73c3a 100644
--- a/be/src/runtime/workload_management/workload_action.h
+++ b/be/src/runtime/workload_management/workload_action.h
@@ -55,9 +55,7 @@ private:
class WorkloadActionFactory {
public:
static std::unique_ptr<WorkloadAction> create_workload_action(TWorkloadAction* action) {
- if (TWorkloadActionType::type::MOVE_QUERY_TO_GROUP == action->action) {
- return std::make_unique<WorkloadActionMoveQuery>(action->action_args);
- } else if (TWorkloadActionType::type::CANCEL_QUERY == action->action) {
+ if (TWorkloadActionType::type::CANCEL_QUERY == action->action) {
return std::make_unique<WorkloadActionCancelQuery>();
}
LOG(ERROR) << "not find a action " << action->action;
diff --git a/be/src/runtime/workload_management/workload_condition.h b/be/src/runtime/workload_management/workload_condition.h
index 3486742c968..96387a2af41 100644
--- a/be/src/runtime/workload_management/workload_condition.h
+++ b/be/src/runtime/workload_management/workload_condition.h
@@ -84,9 +84,9 @@ public:
TWorkloadMetricType::type metric_name = t_cond->metric_name;
if (TWorkloadMetricType::type::QUERY_TIME == metric_name) {
return std::make_unique<WorkloadConditionQueryTime>(op, str_val);
- } else if (TWorkloadMetricType::type::SCAN_ROWS == metric_name) {
+ } else if (TWorkloadMetricType::type::BE_SCAN_ROWS == metric_name) {
return std::make_unique<WorkloadConditionScanRows>(op, str_val);
- } else if (TWorkloadMetricType::type::SCAN_BYTES == metric_name) {
+ } else if (TWorkloadMetricType::type::BE_SCAN_BYTES == metric_name) {
return std::make_unique<WorkloadConditionScanBytes>(op, str_val);
}
LOG(ERROR) << "not find a metric name " << metric_name;
diff --git a/be/src/runtime/workload_management/workload_sched_policy_mgr.cpp b/be/src/runtime/workload_management/workload_sched_policy_mgr.cpp
index 731dd0c8661..c41a9e723e3 100644
--- a/be/src/runtime/workload_management/workload_sched_policy_mgr.cpp
+++ b/be/src/runtime/workload_management/workload_sched_policy_mgr.cpp
@@ -18,6 +18,7 @@
#include "runtime/workload_management/workload_sched_policy_mgr.h"
#include "runtime/fragment_mgr.h"
+#include "runtime/runtime_query_statistics_mgr.h"
namespace doris {
@@ -76,6 +77,7 @@ void WorkloadSchedPolicyMgr::_schedule_workload() {
while (!_stop_latch.wait_for(std::chrono::milliseconds(500))) {
// 1 get query info
std::vector<WorkloadQueryInfo> list;
+ //todo(wb) maybe we can get runtime queryinfo from RuntimeQueryStatiticsMgr directly
_exec_env->fragment_mgr()->get_runtime_query_info(&list);
// todo: add timer
if (list.size() == 0) {
@@ -84,6 +86,9 @@ void WorkloadSchedPolicyMgr::_schedule_workload() {
for (int i = 0; i < list.size(); i++) {
WorkloadQueryInfo* query_info_ptr = &(list[i]);
+ _exec_env->runtime_query_statistics_mgr()->get_metric_map(query_info_ptr->query_id,
+ query_info_ptr->metric_map);
+
// 2 get matched policy
std::map<WorkloadActionType, std::shared_ptr<WorkloadSchedPolicy>> matched_policy_map;
{
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 541592f2095..0d89becbb28 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2391,9 +2391,6 @@ public class Config extends ConfigBase {
@ConfField(mutable = true)
public static int workload_sched_policy_interval_ms = 10000; // 10s
- @ConfField(mutable = true)
- public static int workload_action_interval_ms = 10000; // 10s
-
@ConfField(mutable = true, masterOnly = true)
public static int workload_max_policy_num = 25;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 8c65a71dafa..76ff70589b9 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -111,7 +111,6 @@ import org.apache.doris.common.io.CountingDataOutputStream;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.publish.TopicPublisher;
import org.apache.doris.common.publish.TopicPublisherThread;
-import org.apache.doris.common.publish.WorkloadActionPublishThread;
import org.apache.doris.common.publish.WorkloadGroupPublisher;
import org.apache.doris.common.util.Daemon;
import org.apache.doris.common.util.DynamicPartitionUtil;
@@ -522,8 +521,6 @@ public class Env {
private TopicPublisherThread topicPublisherThread;
- private WorkloadActionPublishThread workloadActionPublisherThread;
-
private MTMVService mtmvService;
private InsertOverwriteManager insertOverwriteManager;
@@ -761,8 +758,6 @@ public class Env {
this.queryCancelWorker = new QueryCancelWorker(systemInfo);
this.topicPublisherThread = new TopicPublisherThread(
"TopicPublisher", Config.publish_topic_info_interval_ms, systemInfo);
- this.workloadActionPublisherThread = new WorkloadActionPublishThread("WorkloadActionPublisher",
- Config.workload_action_interval_ms, systemInfo);
this.mtmvService = new MTMVService();
this.insertOverwriteManager = new InsertOverwriteManager();
}
@@ -1042,7 +1037,6 @@ public class Env {
workloadGroupMgr.startUpdateThread();
workloadSchedPolicyMgr.start();
- workloadActionPublisherThread.start();
workloadRuntimeStatusMgr.start();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/publish/WorkloadActionPublishThread.java b/fe/fe-core/src/main/java/org/apache/doris/common/publish/WorkloadActionPublishThread.java
deleted file mode 100644
index cacd7a9da9f..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/common/publish/WorkloadActionPublishThread.java
+++ /dev/null
@@ -1,123 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.common.publish;
-
-import org.apache.doris.common.ClientPool;
-import org.apache.doris.common.ThreadPoolManager;
-import org.apache.doris.common.util.Daemon;
-import org.apache.doris.system.Backend;
-import org.apache.doris.system.SystemInfoService;
-import org.apache.doris.thrift.BackendService;
-import org.apache.doris.thrift.TNetworkAddress;
-import org.apache.doris.thrift.TPublishTopicRequest;
-import org.apache.doris.thrift.TTopicInfoType;
-import org.apache.doris.thrift.TopicInfo;
-
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ExecutorService;
-
-public class WorkloadActionPublishThread extends Daemon {
-
- private ExecutorService executor = ThreadPoolManager
- .newDaemonFixedThreadPool(4, 256, "workload-action-publish-thread", true);
-
- private static final Logger LOG = LogManager.getLogger(WorkloadActionPublishThread.class);
-
- public static Map<TTopicInfoType, List<TopicInfo>> workloadActionToplicInfoMap
- = new HashMap<TTopicInfoType, List<TopicInfo>>();
-
- public static synchronized void putWorkloadAction(TTopicInfoType type, TopicInfo topicInfo) {
- List<TopicInfo> list = workloadActionToplicInfoMap.get(type);
- if (list == null) {
- list = new ArrayList<TopicInfo>();
- workloadActionToplicInfoMap.put(type, list);
- }
- list.add(topicInfo);
- }
-
- public static synchronized Map<TTopicInfoType, List<TopicInfo>> getCurrentWorkloadActionMap() {
- Map<TTopicInfoType, List<TopicInfo>> retMap = workloadActionToplicInfoMap;
- workloadActionToplicInfoMap = new HashMap<TTopicInfoType, List<TopicInfo>>();
- return retMap;
- }
-
- private SystemInfoService clusterInfoService;
-
- public WorkloadActionPublishThread(String name, long intervalMs,
- SystemInfoService clusterInfoService) {
- super(name, intervalMs);
- this.clusterInfoService = clusterInfoService;
- }
-
- @Override
- protected final void runOneCycle() {
- Map<TTopicInfoType, List<TopicInfo>> actionMap
- = WorkloadActionPublishThread.getCurrentWorkloadActionMap();
- if (actionMap.size() == 0) {
- LOG.info("no workload action found, skip publish");
- return;
- }
- Collection<Backend> currentBeToPublish = clusterInfoService.getIdToBackend().values();
- AckResponseHandler handler = new AckResponseHandler(currentBeToPublish);
- TPublishTopicRequest request = new TPublishTopicRequest();
- request.setTopicMap(actionMap);
- for (Backend be : currentBeToPublish) {
- executor.submit(new WorkloadMoveActionTask(request, be, handler));
- }
- }
-
- public class WorkloadMoveActionTask implements Runnable {
-
- private TPublishTopicRequest request;
-
- private Backend be;
-
- private ResponseHandler handler;
-
- public WorkloadMoveActionTask(TPublishTopicRequest request, Backend be,
- ResponseHandler handler) {
- this.request = request;
- this.be = be;
- this.handler = handler;
- }
-
- @Override
- public void run() {
- long beginTime = System.currentTimeMillis();
- try {
- TNetworkAddress addr = new TNetworkAddress(be.getHost(), be.getBePort());
- BackendService.Client client = ClientPool.backendPool.borrowObject(addr);
- client.publishTopicInfo(request);
- LOG.info("publish move action topic to be {} success, time cost={} ms",
- be.getHost(), (System.currentTimeMillis() - beginTime));
- } catch (Exception e) {
- LOG.warn("publish move action topic to be {} error happens: , time cost={} ms",
- be.getHost(), (System.currentTimeMillis() - beginTime), e);
- } finally {
- handler.onResponse(be);
- }
- }
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadAction.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadAction.java
index d9298781476..661ea6a45fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadAction.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadAction.java
@@ -30,8 +30,6 @@ public interface WorkloadAction {
throws UserException {
if (WorkloadActionType.CANCEL_QUERY.equals(workloadActionMeta.action)) {
return WorkloadActionCancelQuery.createWorkloadAction();
- } else if (WorkloadActionType.MOVE_QUERY_TO_GROUP.equals(workloadActionMeta.action)) {
- return WorkloadActionMoveQueryToGroup.createWorkloadAction(workloadActionMeta.actionArgs);
} else if (WorkloadActionType.SET_SESSION_VARIABLE.equals(workloadActionMeta.action)) {
return WorkloadActionSetSessionVar.createWorkloadAction(workloadActionMeta.actionArgs);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadActionMoveQueryToGroup.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadActionMoveQueryToGroup.java
deleted file mode 100644
index 59e09345fa9..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadActionMoveQueryToGroup.java
+++ /dev/null
@@ -1,67 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.resource.workloadschedpolicy;
-
-import org.apache.doris.common.publish.WorkloadActionPublishThread;
-import org.apache.doris.qe.QeProcessorImpl;
-import org.apache.doris.thrift.TTopicInfoType;
-import org.apache.doris.thrift.TWorkloadMoveQueryToGroupAction;
-import org.apache.doris.thrift.TopicInfo;
-
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-
-public class WorkloadActionMoveQueryToGroup implements WorkloadAction {
-
- private static final Logger LOG = LogManager.getLogger(WorkloadActionMoveQueryToGroup.class);
-
- private long dstWgId;
-
- public WorkloadActionMoveQueryToGroup(long dstWgId) {
- this.dstWgId = dstWgId;
- }
-
- @Override
- public void exec(WorkloadQueryInfo queryInfo) {
- if (queryInfo.context != null && !queryInfo.context.isKilled()
- && queryInfo.tUniqueId != null
- && QeProcessorImpl.INSTANCE.getCoordinator(queryInfo.tUniqueId) != null) {
- LOG.info("try move query {} to group {}", queryInfo.queryId, dstWgId);
-
- TWorkloadMoveQueryToGroupAction moveQueryToGroupAction = new TWorkloadMoveQueryToGroupAction();
- moveQueryToGroupAction.setQueryId(queryInfo.tUniqueId);
- moveQueryToGroupAction.setWorkloadGroupId(dstWgId);
-
- TopicInfo topicInfo = new TopicInfo();
- topicInfo.setMoveAction(moveQueryToGroupAction);
-
- WorkloadActionPublishThread.putWorkloadAction(TTopicInfoType.MOVE_QUERY_TO_GROUP, topicInfo);
- }
- }
-
- @Override
- public WorkloadActionType getWorkloadActionType() {
- return WorkloadActionType.MOVE_QUERY_TO_GROUP;
- }
-
- public static WorkloadActionMoveQueryToGroup createWorkloadAction(String groupId) {
- long wgId = Long.parseLong(groupId);
- return new WorkloadActionMoveQueryToGroup(wgId);
- }
-
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
index 1f75d81794f..5d89d2afae9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
@@ -33,6 +33,10 @@ public interface WorkloadCondition {
return WorkloadConditionUsername.createWorkloadCondition(cm.op, cm.value);
} else if (WorkloadMetricType.QUERY_TIME.equals(cm.metricName)) {
return WorkloadConditionQueryTime.createWorkloadCondition(cm.op, cm.value);
+ } else if (WorkloadMetricType.BE_SCAN_ROWS.equals(cm.metricName)) {
+ return WorkloadConditionBeScanRows.createWorkloadCondition(cm.op, cm.value);
+ } else if (WorkloadMetricType.BE_SCAN_BYTES.equals(cm.metricName)) {
+ return WorkloadConditionBeScanBytes.createWorkloadCondition(cm.op, cm.value);
}
throw new UserException("invalid metric name:" + cm.metricName);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionBeScanBytes.java
similarity index 51%
copy from fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
copy to fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionBeScanBytes.java
index 1f75d81794f..7431f2e0c4f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionBeScanBytes.java
@@ -19,22 +19,34 @@ package org.apache.doris.resource.workloadschedpolicy;
import org.apache.doris.common.UserException;
+public class WorkloadConditionBeScanBytes implements WorkloadCondition {
-public interface WorkloadCondition {
+ private long value;
- boolean eval(String strValue);
+ private WorkloadConditionOperator op;
- WorkloadMetricType getMetricType();
+ public WorkloadConditionBeScanBytes(WorkloadConditionOperator op, long value) {
+ this.op = op;
+ this.value = value;
+ }
+
+ @Override
+ public boolean eval(String strValue) {
+ // currently not support run in fe, so this condition never match
+ return false;
+ }
- // NOTE(wb) currently createPolicyCondition is also used when replay meta, it better not contains heavy check
- static WorkloadCondition createWorkloadCondition(WorkloadConditionMeta cm)
+ public static WorkloadConditionBeScanBytes createWorkloadCondition(WorkloadConditionOperator op, String value)
throws UserException {
- if (WorkloadMetricType.USERNAME.equals(cm.metricName)) {
- return WorkloadConditionUsername.createWorkloadCondition(cm.op, cm.value);
- } else if (WorkloadMetricType.QUERY_TIME.equals(cm.metricName)) {
- return WorkloadConditionQueryTime.createWorkloadCondition(cm.op, cm.value);
+ long longValue = Long.parseLong(value);
+ if (longValue < 0) {
+ throw new UserException("invalid scan bytes value, " + longValue + ", it requires >= 0");
}
- throw new UserException("invalid metric name:" + cm.metricName);
+ return new WorkloadConditionBeScanBytes(op, longValue);
}
+ @Override
+ public WorkloadMetricType getMetricType() {
+ return WorkloadMetricType.BE_SCAN_BYTES;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionBeScanRows.java
similarity index 51%
copy from fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
copy to fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionBeScanRows.java
index 1f75d81794f..c2fb638e082 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadCondition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionBeScanRows.java
@@ -19,22 +19,34 @@ package org.apache.doris.resource.workloadschedpolicy;
import org.apache.doris.common.UserException;
+public class WorkloadConditionBeScanRows implements WorkloadCondition {
-public interface WorkloadCondition {
+ private long value;
- boolean eval(String strValue);
+ private WorkloadConditionOperator op;
- WorkloadMetricType getMetricType();
+ public WorkloadConditionBeScanRows(WorkloadConditionOperator op, long value) {
+ this.op = op;
+ this.value = value;
+ }
+
+ @Override
+ public boolean eval(String strValue) {
+ // currently not support run in fe, so this condition never match
+ return false;
+ }
- // NOTE(wb) currently createPolicyCondition is also used when replay meta, it better not contains heavy check
- static WorkloadCondition createWorkloadCondition(WorkloadConditionMeta cm)
+ public static WorkloadConditionBeScanRows createWorkloadCondition(WorkloadConditionOperator op, String value)
throws UserException {
- if (WorkloadMetricType.USERNAME.equals(cm.metricName)) {
- return WorkloadConditionUsername.createWorkloadCondition(cm.op, cm.value);
- } else if (WorkloadMetricType.QUERY_TIME.equals(cm.metricName)) {
- return WorkloadConditionQueryTime.createWorkloadCondition(cm.op, cm.value);
+ long longValue = Long.parseLong(value);
+ if (longValue < 0) {
+ throw new UserException("invalid scan rows value, " + longValue + ", it requires >= 0");
}
- throw new UserException("invalid metric name:" + cm.metricName);
+ return new WorkloadConditionBeScanRows(op, longValue);
}
+ @Override
+ public WorkloadMetricType getMetricType() {
+ return WorkloadMetricType.BE_SCAN_ROWS;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionMeta.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionMeta.java
index d5d2f922f3f..52f50f924fc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionMeta.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadConditionMeta.java
@@ -44,6 +44,10 @@ public class WorkloadConditionMeta {
return WorkloadMetricType.USERNAME;
} else if (WorkloadMetricType.QUERY_TIME.toString().equalsIgnoreCase(metricStr)) {
return WorkloadMetricType.QUERY_TIME;
+ } else if (WorkloadMetricType.BE_SCAN_ROWS.toString().equalsIgnoreCase(metricStr)) {
+ return WorkloadMetricType.BE_SCAN_ROWS;
+ } else if (WorkloadMetricType.BE_SCAN_BYTES.toString().equalsIgnoreCase(metricStr)) {
+ return WorkloadMetricType.BE_SCAN_BYTES;
}
throw new UserException("invalid metric name:" + metricStr);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadMetricType.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadMetricType.java
index f81d75c675f..ed17414ec45 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadMetricType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadMetricType.java
@@ -18,5 +18,5 @@
package org.apache.doris.resource.workloadschedpolicy;
public enum WorkloadMetricType {
- USERNAME, QUERY_TIME, SCAN_ROWS, SCAN_BYTES
+ USERNAME, QUERY_TIME, BE_SCAN_ROWS, BE_SCAN_BYTES
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicy.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicy.java
index 13637b5b8be..8b028e3e75f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicy.java
@@ -52,7 +52,9 @@ public class WorkloadSchedPolicy implements Writable, GsonPostProcessable {
// used for convert fe type to thrift type
private static ImmutableMap<WorkloadMetricType, TWorkloadMetricType> METRIC_MAP
= new ImmutableMap.Builder<WorkloadMetricType, TWorkloadMetricType>()
- .put(WorkloadMetricType.QUERY_TIME, TWorkloadMetricType.QUERY_TIME).build();
+ .put(WorkloadMetricType.QUERY_TIME, TWorkloadMetricType.QUERY_TIME)
+ .put(WorkloadMetricType.BE_SCAN_ROWS, TWorkloadMetricType.BE_SCAN_ROWS)
+ .put(WorkloadMetricType.BE_SCAN_BYTES, TWorkloadMetricType.BE_SCAN_BYTES).build();
private static ImmutableMap<WorkloadActionType, TWorkloadActionType> ACTION_MAP
= new ImmutableMap.Builder<WorkloadActionType, TWorkloadActionType>()
.put(WorkloadActionType.MOVE_QUERY_TO_GROUP, TWorkloadActionType.MOVE_QUERY_TO_GROUP)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicyMgr.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicyMgr.java
index 0074020735e..5c35d1ee095 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicyMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadschedpolicy/WorkloadSchedPolicyMgr.java
@@ -86,9 +86,8 @@ public class WorkloadSchedPolicyMgr implements Writable, GsonPostProcessable {
.add(WorkloadActionType.CANCEL_QUERY).build();
public static final ImmutableSet<WorkloadMetricType> BE_METRIC_SET
- = new ImmutableSet.Builder<WorkloadMetricType>().add(WorkloadMetricType.SCAN_ROWS)
- .add(WorkloadMetricType.SCAN_BYTES).add(WorkloadMetricType.QUERY_TIME)
- .build();
+ = new ImmutableSet.Builder<WorkloadMetricType>().add(WorkloadMetricType.BE_SCAN_ROWS)
+ .add(WorkloadMetricType.BE_SCAN_BYTES).add(WorkloadMetricType.QUERY_TIME).build();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
@@ -122,15 +121,12 @@ public class WorkloadSchedPolicyMgr implements Writable, GsonPostProcessable {
}
String username = cctx.getQualifiedUser();
- long queryTime = System.currentTimeMillis() - cctx.getStartTime();
-
WorkloadQueryInfo policyQueryInfo = new WorkloadQueryInfo();
policyQueryInfo.queryId = cctx.queryId() == null ? null : DebugUtil.printId(cctx.queryId());
policyQueryInfo.tUniqueId = cctx.queryId();
policyQueryInfo.context = cctx;
policyQueryInfo.metricMap = new HashMap<>();
policyQueryInfo.metricMap.put(WorkloadMetricType.USERNAME, username);
- policyQueryInfo.metricMap.put(WorkloadMetricType.QUERY_TIME, String.valueOf(queryTime));
queryInfoList.add(policyQueryInfo);
}
@@ -174,19 +170,7 @@ public class WorkloadSchedPolicyMgr implements Writable, GsonPostProcessable {
List<WorkloadActionMeta> originActions = createStmt.getActions();
List<WorkloadAction> policyActionList = new ArrayList<>();
for (WorkloadActionMeta workloadActionMeta : originActions) {
- WorkloadActionType actionName = workloadActionMeta.action;
- String actionArgs = workloadActionMeta.actionArgs;
-
- // we need convert wgName to wgId, because wgName may change
- if (WorkloadActionType.MOVE_QUERY_TO_GROUP.equals(actionName)) {
- Long wgId = Env.getCurrentEnv().getWorkloadGroupMgr().getWorkloadGroupIdByName(actionArgs);
- if (wgId == null) {
- throw new UserException(
- "can not find workload group " + actionArgs + " when set workload sched policy");
- }
- workloadActionMeta.actionArgs = wgId.toString();
- }
-
+ // todo(wb) support move action
WorkloadAction ret = WorkloadAction.createWorkloadAction(workloadActionMeta);
policyActionList.add(ret);
}
diff --git a/gensrc/thrift/BackendService.thrift b/gensrc/thrift/BackendService.thrift
index c59abd65d3b..24edaefc103 100644
--- a/gensrc/thrift/BackendService.thrift
+++ b/gensrc/thrift/BackendService.thrift
@@ -179,15 +179,10 @@ struct TWorkloadGroupInfo {
9: optional i32 scan_thread_num
}
-struct TWorkloadMoveQueryToGroupAction {
- 1: optional Types.TUniqueId query_id
- 2: optional i64 workload_group_id
-}
-
enum TWorkloadMetricType {
QUERY_TIME
- SCAN_ROWS
- SCAN_BYTES
+ BE_SCAN_ROWS
+ BE_SCAN_BYTES
}
enum TCompareOperator {
@@ -226,8 +221,7 @@ struct TWorkloadSchedPolicy {
struct TopicInfo {
1: optional TWorkloadGroupInfo workload_group_info
- 2: optional TWorkloadMoveQueryToGroupAction move_action
- 3: optional TWorkloadSchedPolicy workload_sched_policy
+ 2: optional TWorkloadSchedPolicy workload_sched_policy
}
struct TPublishTopicRequest {
diff --git a/regression-test/data/workload_manager_p0/test_workload_sched_policy.out b/regression-test/data/workload_manager_p0/test_workload_sched_policy.out
index 0c86700866d..d32fff321e4 100644
--- a/regression-test/data/workload_manager_p0/test_workload_sched_policy.out
+++ b/regression-test/data/workload_manager_p0/test_workload_sched_policy.out
@@ -2,7 +2,6 @@
-- !select_policy_tvf --
be_policy query_time > 10 cancel_query 10 false 0
fe_policy username = root set_session_variable "workload_group=normal" 10 false 0
-move_action_policy query_time > 10 move_query_to_group "normal" 0 true 0
set_action_policy username = root set_session_variable "workload_group=normal" 0 true 0
test_cancel_policy query_time > 10 cancel_query 0 false 0
diff --git a/regression-test/suites/workload_manager_p0/test_workload_sched_policy.groovy b/regression-test/suites/workload_manager_p0/test_workload_sched_policy.groovy
index cabba8ee008..603bbdf520e 100644
--- a/regression-test/suites/workload_manager_p0/test_workload_sched_policy.groovy
+++ b/regression-test/suites/workload_manager_p0/test_workload_sched_policy.groovy
@@ -20,7 +20,6 @@ suite("test_workload_sched_policy") {
sql "set experimental_enable_nereids_planner = false;"
sql "drop workload schedule policy if exists test_cancel_policy;"
- sql "drop workload schedule policy if exists move_action_policy;"
sql "drop workload schedule policy if exists set_action_policy;"
sql "drop workload schedule policy if exists fe_policy;"
sql "drop workload schedule policy if exists be_policy;"
@@ -30,17 +29,12 @@ suite("test_workload_sched_policy") {
" conditions(query_time > 10) " +
" actions(cancel_query) properties('enabled'='false'); "
- // 2 create cancel policy
- sql "create workload schedule policy move_action_policy " +
- "conditions(query_time > 10) " +
- "actions(move_query_to_group 'normal');"
-
- // 3 create set policy
+ // 2 create set policy
sql "create workload schedule policy set_action_policy " +
"conditions(username='root') " +
"actions(set_session_variable 'workload_group=normal');"
- // 4 create policy run in fe
+ // 3 create policy run in fe
sql "create workload schedule policy fe_policy " +
"conditions(username='root') " +
"actions(set_session_variable 'workload_group=normal') " +
@@ -49,7 +43,7 @@ suite("test_workload_sched_policy") {
"'priority'='10' " +
");"
- // 5 create policy run in be
+ // 4 create policy run in be
sql "create workload schedule policy be_policy " +
"conditions(query_time > 10) " +
"actions(cancel_query) " +
@@ -96,14 +90,6 @@ suite("test_workload_sched_policy") {
exception "priority can only between"
}
- test {
- sql "create workload schedule policy conflict_policy " +
- "conditions (query_time > 0)" +
- "actions(cancel_query, move_query_to_group 'normal');"
-
- exception "can not exist in one policy at same time"
- }
-
test {
sql "create workload schedule policy conflict_policy " +
"conditions (query_time > 0) " +
@@ -122,7 +108,6 @@ suite("test_workload_sched_policy") {
// drop
sql "drop workload schedule policy test_cancel_policy;"
- sql "drop workload schedule policy move_action_policy;"
sql "drop workload schedule policy set_action_policy;"
sql "drop workload schedule policy fe_policy;"
sql "drop workload schedule policy be_policy;"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/06: [feature](doris compose) Add create cloud cluster (#31315)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3ba4fcdccc2d34cb31ceef5a66fc1731d7e2132f
Author: yujun <yu...@gmail.com>
AuthorDate: Mon Feb 26 17:26:02 2024 +0800
[feature](doris compose) Add create cloud cluster (#31315)
---
docker/runtime/doris-compose/Dockerfile | 19 +-
docker/runtime/doris-compose/Readme.md | 31 +-
docker/runtime/doris-compose/cluster.py | 326 ++++++++++++++----
docker/runtime/doris-compose/command.py | 365 +++++++++++++++++----
docker/runtime/doris-compose/doris-compose.py | 2 +-
.../doris-compose/resource/cloud.ini.example | 25 ++
docker/runtime/doris-compose/resource/common.sh | 95 +++++-
docker/runtime/doris-compose/resource/fdb.conf | 45 +++
docker/runtime/doris-compose/resource/init_be.sh | 121 +++++--
.../runtime/doris-compose/resource/init_cloud.sh | 104 ++++++
.../resource/{common.sh => init_fdb.sh} | 65 ++--
docker/runtime/doris-compose/resource/init_fe.sh | 104 +++++-
.../org/apache/doris/regression/suite/Suite.groovy | 17 +-
.../doris/regression/suite/SuiteCluster.groovy | 16 +-
.../suites/demo_p0/docker_action.groovy | 15 +
15 files changed, 1151 insertions(+), 199 deletions(-)
diff --git a/docker/runtime/doris-compose/Dockerfile b/docker/runtime/doris-compose/Dockerfile
index a524ce8c189..2306bf67cd2 100644
--- a/docker/runtime/doris-compose/Dockerfile
+++ b/docker/runtime/doris-compose/Dockerfile
@@ -25,12 +25,13 @@ ARG OUT_DIRECTORY=output
ENV JAVA_HOME="/usr/local/openjdk-8/"
ENV jacoco_version 0.8.8
+RUN mkdir -p /opt/apache-doris/coverage
+
RUN sed -i s@/deb.debian.org/@/mirrors.aliyun.com/@g /etc/apt/sources.list
RUN apt-get clean
-
RUN apt-get update && \
- apt-get install -y default-mysql-client python lsof tzdata curl unzip && \
+ apt-get install -y default-mysql-client python lsof tzdata curl unzip patchelf jq && \
ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
dpkg-reconfigure -f noninteractive tzdata && \
apt-get clean
@@ -39,7 +40,17 @@ RUN curl -f https://repo1.maven.org/maven2/org/jacoco/jacoco/$jacoco_version/jac
mkdir /jacoco && \
unzip jacoco.zip -d /jacoco
-ADD $OUT_DIRECTORY /opt/apache-doris/
-RUN mkdir -p /opt/apache-doris/coverage
+# cloud
+COPY ${OUT_DIRECTORY}/../cloud/CMakeLists.txt ${OUT_DIRECTORY}/../cloud/output* /opt/apache-doris/cloud/
+RUN <<EOF
+ mkdir /opt/apache-doris/fdb
+ if [ -d /opt/apache-doris/cloud/bin ]; then
+ sed -i 's/\<chmod\>/echo/g' /opt/apache-doris/cloud/bin/start.sh
+ fi
+EOF
+
+# fe and be
+COPY $OUT_DIRECTORY /opt/apache-doris/
# in docker, run 'chmod 755 doris_be' first time cost 1min, remove it.
RUN sed -i 's/\<chmod\>/echo/g' /opt/apache-doris/be/bin/start_be.sh
+
diff --git a/docker/runtime/doris-compose/Readme.md b/docker/runtime/doris-compose/Readme.md
index 7e7173e7d28..cd3d7805fe8 100644
--- a/docker/runtime/doris-compose/Readme.md
+++ b/docker/runtime/doris-compose/Readme.md
@@ -26,10 +26,13 @@ Use doris compose to create doris docker compose clusters.
1. The doris image should contains:
```
-/opt/apache-doris/{fe, be}
+/opt/apache-doris/{fe, be, cloud}
```
-if build doris use `sh build.sh`, then its output satisfy with this, then run command in doris root
+if don't create cloud cluster, the image no need to contains the cloud pkg.
+
+
+if build doris use `sh build.sh --fe --be --cloud`, then its output satisfy with all above, then run command in doris root
```
docker build -f docker/runtime/doris-compose/Dockerfile -t <image> .
@@ -52,13 +55,29 @@ python -m pip install --user -r docker/runtime/doris-compose/requirements.txt
python docker/runtime/doris-compose/doris-compose.py up <cluster-name> <image?>
--add-fe-num <add-fe-num> --add-be-num <add-be-num>
--fe-id <fd-id> --be-id <be-id>
-
+ ...
+ [ --cloud ]
```
if it's a new cluster, must specific the image.
add fe/be nodes with the specific image, or update existing nodes with `--fe-id`, `--be-id`
+
+For create a cloud cluster, steps are as below:
+1. Write cloud s3 store config file, its default path is '/tmp/doris/cloud.ini'.
+ It's defined in environment variable DORIS_CLOUD_CFG_FILE, user can change this env var to change its path.
+ A Example file is locate in 'docker/runtime/doris-compose/resource/cloud.ini.example'.
+2. Use doris compose up command with option '--cloud' to create a new cloud cluster.
+
+The simplest way to create a cloud cluster:
+
+```
+python docker/runtime/doris-compose/doris-compose.py up <cluster-name> <image> --cloud
+```
+
+It will create 1 fdb, 1 meta service server, 1 recycler, 3 fe and 3 be.
+
### Remove node from the cluster
```
@@ -96,5 +115,11 @@ There are more options about doris-compose. Just try
python docker/runtime/doris-compose/doris-compose.py <command> -h
```
+### Generate regression custom conf file
+
+```
+python docker/runtime/doris-compose/doris-compose.py config <cluster-name>
+```
+Generate regression-conf-custom.groovy to connect to the specific docker cluster.
diff --git a/docker/runtime/doris-compose/cluster.py b/docker/runtime/doris-compose/cluster.py
index 23016bc27ba..660a243f5c4 100644
--- a/docker/runtime/doris-compose/cluster.py
+++ b/docker/runtime/doris-compose/cluster.py
@@ -28,6 +28,8 @@ DORIS_SUBNET_START = int(os.getenv("DORIS_SUBNET_START", 128))
LOCAL_RESOURCE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"resource")
DOCKER_RESOURCE_PATH = os.path.join(DOCKER_DORIS_PATH, "resource")
+CLOUD_CFG_FILE = os.getenv("DORIS_CLOUD_CFG_FILE",
+ os.path.join(LOCAL_DORIS_PATH, 'cloud.ini'))
FE_HTTP_PORT = 8030
FE_RPC_PORT = 9020
@@ -39,6 +41,10 @@ BE_WEBSVR_PORT = 8040
BE_HEARTBEAT_PORT = 9050
BE_BRPC_PORT = 8060
+FDB_PORT = 4500
+
+MS_PORT = 5000
+
ID_LIMIT = 10000
IP_PART4_SIZE = 200
@@ -99,6 +105,24 @@ def get_master_fe_endpoint(cluster_name):
return ""
+def get_node_seq(node_type, id):
+ seq = id
+ seq += IP_PART4_SIZE
+ if node_type == Node.TYPE_FE:
+ seq += 0 * ID_LIMIT
+ elif node_type == Node.TYPE_BE:
+ seq += 1 * ID_LIMIT
+ elif node_type == Node.TYPE_MS:
+ seq += 2 * ID_LIMIT
+ elif node_type == Node.TYPE_RECYCLE:
+ seq += 3 * ID_LIMIT
+ elif node_type == Node.TYPE_FDB:
+ seq += 4 * ID_LIMIT
+ else:
+ seq += 5 * ID_LIMIT
+ return seq
+
+
class NodeMeta(object):
def __init__(self, image):
@@ -151,46 +175,61 @@ class Group(object):
class Node(object):
TYPE_FE = "fe"
TYPE_BE = "be"
- TYPE_ALL = [TYPE_FE, TYPE_BE]
+ TYPE_MS = "ms"
+ TYPE_RECYCLE = "recycle"
+ TYPE_FDB = "fdb"
+ TYPE_ALL = [TYPE_FE, TYPE_BE, TYPE_MS, TYPE_RECYCLE, TYPE_FDB]
- def __init__(self, cluster_name, coverage_dir, id, subnet, meta):
- self.cluster_name = cluster_name
- self.coverage_dir = coverage_dir
+ def __init__(self, cluster, id, meta):
+ self.cluster = cluster
self.id = id
- self.subnet = subnet
self.meta = meta
@staticmethod
- def new(cluster_name, coverage_dir, node_type, id, subnet, meta):
+ def new(cluster, node_type, id, meta):
if node_type == Node.TYPE_FE:
- return FE(cluster_name, coverage_dir, id, subnet, meta)
+ return FE(cluster, id, meta)
elif node_type == Node.TYPE_BE:
- return BE(cluster_name, coverage_dir, id, subnet, meta)
+ return BE(cluster, id, meta)
+ elif node_type == Node.TYPE_MS:
+ return MS(cluster, id, meta)
+ elif node_type == Node.TYPE_RECYCLE:
+ return RECYCLE(cluster, id, meta)
+ elif node_type == Node.TYPE_FDB:
+ return FDB(cluster, id, meta)
else:
raise Exception("Unknown node type {}".format(node_type))
- def init_conf(self, config):
+ def init(self):
+ self.init_conf()
+
+ def init_conf(self):
path = self.get_path()
os.makedirs(path, exist_ok=True)
+ config = self.get_add_init_config()
+
# copy config to local
conf_dir = os.path.join(path, "conf")
if not os.path.exists(conf_dir) or utils.is_dir_empty(conf_dir):
- utils.copy_image_directory(
- self.get_image(), "{}/{}/conf".format(DOCKER_DORIS_PATH,
- self.node_type()),
- conf_dir)
+ self.copy_conf_to_local(conf_dir)
assert not utils.is_dir_empty(conf_dir), "conf directory {} is empty, " \
"check doris path in image is correct".format(conf_dir)
utils.enable_dir_with_rw_perm(conf_dir)
if config:
with open(os.path.join(conf_dir, self.conf_file_name()),
"a") as f:
+ f.write("\n")
for item in config:
f.write(item + "\n")
for sub_dir in self.expose_sub_dirs():
os.makedirs(os.path.join(path, sub_dir), exist_ok=True)
+ def copy_conf_to_local(self, local_conf_dir):
+ utils.copy_image_directory(self.get_image(),
+ "{}/conf".format(self.docker_home_dir()),
+ local_conf_dir)
+
def is_fe(self):
return self.node_type() == Node.TYPE_FE
@@ -210,7 +249,7 @@ class Node(object):
return "{}-{}".format(self.node_type(), self.id)
def get_path(self):
- return os.path.join(get_cluster_path(self.cluster_name),
+ return os.path.join(get_cluster_path(self.cluster.name),
self.get_name())
def get_image(self):
@@ -220,15 +259,8 @@ class Node(object):
self.meta.image = image
def get_ip(self):
- seq = self.id
- seq += IP_PART4_SIZE
- if self.node_type() == Node.TYPE_FE:
- seq += 0 * ID_LIMIT
- elif self.node_type() == Node.TYPE_BE:
- seq += 1 * ID_LIMIT
- else:
- seq += 2 * ID_LIMIT
- return "{}.{}.{}".format(self.subnet, int(seq / IP_PART4_SIZE),
+ seq = get_node_seq(self.node_type(), self.id)
+ return "{}.{}.{}".format(self.cluster.subnet, int(seq / IP_PART4_SIZE),
seq % IP_PART4_SIZE)
@staticmethod
@@ -244,25 +276,30 @@ class Node(object):
return seq
def service_name(self):
- return utils.with_doris_prefix("{}-{}".format(self.cluster_name,
+ return utils.with_doris_prefix("{}-{}".format(self.cluster.name,
self.get_name()))
def docker_env(self):
- enable_coverage = self.coverage_dir
+ enable_coverage = self.cluster.coverage_dir
envs = {
"MY_IP": self.get_ip(),
"MY_ID": self.id,
+ "MY_TYPE": self.node_type(),
"FE_QUERY_PORT": FE_QUERY_PORT,
"FE_EDITLOG_PORT": FE_EDITLOG_PORT,
"BE_HEARTBEAT_PORT": BE_HEARTBEAT_PORT,
- "DORIS_HOME": os.path.join(DOCKER_DORIS_PATH, self.node_type()),
+ "DORIS_HOME": os.path.join(self.docker_home_dir()),
"STOP_GRACE": 1 if enable_coverage else 0,
+ "IS_CLOUD": 1 if self.cluster.is_cloud else 0,
}
+ if self.cluster.is_cloud:
+ envs["META_SERVICE_ENDPOINT"] = self.cluster.get_meta_server_addr()
+
if enable_coverage:
outfile = "{}/coverage/{}-coverage-{}-{}".format(
- DOCKER_DORIS_PATH, self.node_type(), self.cluster_name,
+ DOCKER_DORIS_PATH, self.node_type(), self.cluster.name,
self.id)
if self.node_type() == Node.TYPE_FE:
envs["JACOCO_COVERAGE_OPT"] = "-javaagent:/jacoco/lib/jacocoagent.jar" \
@@ -274,36 +311,41 @@ class Node(object):
return envs
+ def get_add_init_config(self):
+ return []
+
def docker_ports(self):
raise Exception("No implemented")
+ def docker_home_dir(self):
+ raise Exception("No implemented")
+
def compose(self):
volumes = [
- "{}:{}/{}/{}".format(os.path.join(self.get_path(), sub_dir),
- DOCKER_DORIS_PATH, self.node_type(), sub_dir)
+ "{}:{}/{}".format(os.path.join(self.get_path(), sub_dir),
+ self.docker_home_dir(), sub_dir)
for sub_dir in self.expose_sub_dirs()
] + [
"{}:{}:ro".format(LOCAL_RESOURCE_PATH, DOCKER_RESOURCE_PATH),
- "{}:{}/{}/status".format(get_status_path(self.cluster_name),
- DOCKER_DORIS_PATH, self.node_type()),
+ "{}:{}/status".format(get_status_path(self.cluster.name),
+ self.docker_home_dir()),
] + [
"{0}:{0}:ro".format(path)
for path in ("/etc/localtime", "/etc/timezone",
"/usr/share/zoneinfo") if os.path.exists(path)
]
- if self.coverage_dir:
- volumes.append("{}:{}/coverage".format(self.coverage_dir,
+ if self.cluster.coverage_dir:
+ volumes.append("{}:{}/coverage".format(self.cluster.coverage_dir,
DOCKER_DORIS_PATH))
- return {
+ content = {
"cap_add": ["SYS_PTRACE"],
"hostname": self.get_name(),
"container_name": self.service_name(),
- "entrypoint": self.entrypoint(),
"environment": self.docker_env(),
"image": self.get_image(),
"networks": {
- utils.with_doris_prefix(self.cluster_name): {
+ utils.with_doris_prefix(self.cluster.name): {
"ipv4_address": self.get_ip(),
}
},
@@ -315,15 +357,49 @@ class Node(object):
"volumes": volumes,
}
+ if self.entrypoint():
+ content["entrypoint"] = self.entrypoint()
+
+ return content
+
class FE(Node):
+ def get_add_init_config(self):
+ cfg = []
+ if self.cluster.fe_config:
+ cfg += self.cluster.fe_config
+ if self.cluster.is_cloud:
+ cfg += [
+ "cloud_unique_id = " + self.cloud_unique_id(),
+ "meta_service_endpoint = {}".format(
+ self.cluster.get_meta_server_addr()),
+ "",
+ "# For regression-test",
+ "ignore_unsupported_properties_in_cloud_mode = true",
+ "merge_on_write_forced_to_false = true",
+ ]
+
+ return cfg
+
+ def docker_env(self):
+ envs = super().docker_env()
+ if self.cluster.is_cloud:
+ envs["CLOUD_UNIQUE_ID"] = self.cloud_unique_id()
+ return envs
+
+ def cloud_unique_id(self):
+ return "sql_server_{}".format(self.id)
+
def entrypoint(self):
return ["bash", os.path.join(DOCKER_RESOURCE_PATH, "init_fe.sh")]
def docker_ports(self):
return [FE_HTTP_PORT, FE_EDITLOG_PORT, FE_RPC_PORT, FE_QUERY_PORT]
+ def docker_home_dir(self):
+ return os.path.join(DOCKER_DORIS_PATH, "fe")
+
def node_type(self):
return Node.TYPE_FE
@@ -333,17 +409,28 @@ class FE(Node):
class BE(Node):
- def entrypoint(self):
- return ["bash", os.path.join(DOCKER_RESOURCE_PATH, "init_be.sh")]
-
- def docker_ports(self):
- return [BE_WEBSVR_PORT, BE_BRPC_PORT, BE_HEARTBEAT_PORT, BE_PORT]
-
- def node_type(self):
- return Node.TYPE_BE
-
- def expose_sub_dirs(self):
- return super().expose_sub_dirs() + ["storage"]
+ def init(self):
+ super().init()
+ if self.cluster.is_cloud:
+ self.init_cluster_name()
+ self.init_disk(self.cluster.be_disks)
+
+ def get_add_init_config(self):
+ cfg = []
+ if self.cluster.be_config:
+ cfg += self.cluster.be_config
+ if self.cluster.is_cloud:
+ cfg += [
+ "cloud_unique_id = " + self.cloud_unique_id(),
+ "meta_service_endpoint = {}".format(
+ self.cluster.get_meta_server_addr()),
+ 'tmp_file_dirs = [ {"path":"./storage/tmp","max_cache_bytes":10240000," "max_upload_bytes":10240000}]',
+ ]
+ return cfg
+
+ def init_cluster_name(self):
+ with open("{}/conf/CLUSTER_NAME".format(self.get_path()), "w") as f:
+ f.write(self.cluster.be_cluster)
def init_disk(self, be_disks):
path = self.get_path()
@@ -379,32 +466,138 @@ class BE(Node):
with open("{}/conf/{}".format(path, self.conf_file_name()), "a") as f:
storage_root_path = ";".join(dir_descs) if dir_descs else '""'
- f.write("storage_root_path = {}\n".format(storage_root_path))
+ f.write("\nstorage_root_path = {}\n".format(storage_root_path))
+
+ def entrypoint(self):
+ return ["bash", os.path.join(DOCKER_RESOURCE_PATH, "init_be.sh")]
+
+ def docker_env(self):
+ envs = super().docker_env()
+ if self.cluster.is_cloud:
+ envs["CLOUD_UNIQUE_ID"] = self.cloud_unique_id()
+ return envs
+
+ def cloud_unique_id(self):
+ return "compute_node_{}".format(self.id)
+
+ def docker_home_dir(self):
+ return os.path.join(DOCKER_DORIS_PATH, "be")
+
+ def docker_ports(self):
+ return [BE_WEBSVR_PORT, BE_BRPC_PORT, BE_HEARTBEAT_PORT, BE_PORT]
+
+ def node_type(self):
+ return Node.TYPE_BE
+
+ def expose_sub_dirs(self):
+ return super().expose_sub_dirs() + ["storage"]
+
+
+class CLOUD(Node):
+
+ def get_add_init_config(self):
+ return ["fdb_cluster = " + self.cluster.get_fdb_cluster()]
+
+ def docker_home_dir(self):
+ return os.path.join(DOCKER_DORIS_PATH, "cloud")
+
+ def docker_ports(self):
+ return [MS_PORT]
+
+ def conf_file_name(self):
+ return "doris_cloud.conf"
+
+
+class MS(CLOUD):
+
+ def entrypoint(self):
+ return [
+ "bash",
+ os.path.join(DOCKER_RESOURCE_PATH, "init_cloud.sh"),
+ "--meta-service"
+ ]
+
+ def node_type(self):
+ return Node.TYPE_MS
+
+ def docker_env(self):
+ envs = super().docker_env()
+ for key, value in self.cluster.cloud_store_config.items():
+ envs[key] = value
+ return envs
+
+
+class RECYCLE(CLOUD):
+
+ def entrypoint(self):
+ return [
+ "bash",
+ os.path.join(DOCKER_RESOURCE_PATH, "init_cloud.sh"), "--recycler"
+ ]
+
+ def node_type(self):
+ return Node.TYPE_RECYCLE
+
+
+class FDB(Node):
+
+ def copy_conf_to_local(self, local_conf_dir):
+ os.makedirs(local_conf_dir, exist_ok=True)
+ with open(os.path.join(LOCAL_RESOURCE_PATH, "fdb.conf"),
+ "r") as read_file:
+ with open(os.path.join(local_conf_dir, self.conf_file_name()),
+ "w") as f:
+ publish_addr = "{}:{}".format(self.get_ip(), FDB_PORT)
+ f.write(read_file.read().replace("${PUBLISH-ADDRESS}",
+ publish_addr))
+
+ with open(os.path.join(local_conf_dir, "fdb.cluster"), "w") as f:
+ f.write(self.cluster.get_fdb_cluster())
+
+ def entrypoint(self):
+ return ["bash", os.path.join(DOCKER_RESOURCE_PATH, "init_fdb.sh")]
+
+ def docker_home_dir(self):
+ return os.path.join(DOCKER_DORIS_PATH, "fdb")
+
+ def docker_ports(self):
+ return [FDB_PORT]
+
+ def node_type(self):
+ return Node.TYPE_FDB
+
+ def expose_sub_dirs(self):
+ return super().expose_sub_dirs() + ["data"]
class Cluster(object):
- def __init__(self, name, subnet, image, fe_config, be_config, be_disks,
- coverage_dir):
+ def __init__(self, name, subnet, image, is_cloud, fe_config, be_config,
+ be_disks, be_cluster, coverage_dir, cloud_store_config):
self.name = name
self.subnet = subnet
self.image = image
+ self.is_cloud = is_cloud
self.fe_config = fe_config
self.be_config = be_config
self.be_disks = be_disks
+ self.be_cluster = be_cluster
self.coverage_dir = coverage_dir
+ self.cloud_store_config = cloud_store_config
self.groups = {
node_type: Group(node_type)
for node_type in Node.TYPE_ALL
}
@staticmethod
- def new(name, image, fe_config, be_config, be_disks, coverage_dir):
+ def new(name, image, is_cloud, fe_config, be_config, be_disks, be_cluster,
+ coverage_dir, cloud_store_config):
os.makedirs(LOCAL_DORIS_PATH, exist_ok=True)
with filelock.FileLock(os.path.join(LOCAL_DORIS_PATH, "lock")):
subnet = gen_subnet_prefix16()
- cluster = Cluster(name, subnet, image, fe_config, be_config,
- be_disks, coverage_dir)
+ cluster = Cluster(name, subnet, image, is_cloud, fe_config,
+ be_config, be_disks, be_cluster, coverage_dir,
+ cloud_store_config)
os.makedirs(cluster.get_path(), exist_ok=True)
os.makedirs(get_status_path(name), exist_ok=True)
cluster._save_meta()
@@ -440,7 +633,9 @@ class Cluster(object):
# cluster's nodes will update image too if cluster update.
def set_image(self, image):
self.image = image
- for _, group in self.groups.items():
+ for node_type, group in self.groups.items():
+ if node_type == Node.TYPE_FDB:
+ continue
for _, node_meta in group.nodes.items():
node_meta.image = image
@@ -458,16 +653,15 @@ class Cluster(object):
meta = group.get_node(id)
if not meta:
raise Exception("No found {} with id {}".format(node_type, id))
- return Node.new(self.name, self.coverage_dir, node_type, id,
- self.subnet, meta)
+ return Node.new(self, node_type, id, meta)
def get_all_nodes(self, node_type):
group = self.groups.get(node_type, None)
if not group:
raise Exception("Unknown node_type: {}".format(node_type))
return [
- Node.new(self.name, self.coverage_dir, node_type, id, self.subnet,
- meta) for id, meta in group.get_all_nodes().items()
+ Node.new(self, node_type, id, meta)
+ for id, meta in group.get_all_nodes().items()
]
def get_all_nodes_num(self):
@@ -481,15 +675,17 @@ class Cluster(object):
id = self.get_group(node_type).add(id, node_meta)
node = self.get_node(node_type, id)
if not os.path.exists(node.get_path()):
- if node.is_fe():
- node.init_conf(self.fe_config)
- elif node.is_be():
- node.init_conf(self.be_config)
- node.init_disk(self.be_disks)
- else:
- node.init_conf([])
+ node.init()
+
return node
+ def get_fdb_cluster(self):
+ return "123456:123456@{}:{}".format(
+ self.get_node(Node.TYPE_FDB, 1).get_ip(), FDB_PORT)
+
+ def get_meta_server_addr(self):
+ return "{}:{}".format(self.get_node(Node.TYPE_MS, 1).get_ip(), MS_PORT)
+
def remove(self, node_type, id):
group = self.get_group(node_type)
group.remove(id)
diff --git a/docker/runtime/doris-compose/command.py b/docker/runtime/doris-compose/command.py
index 261d63ee803..7e1a4ef695a 100644
--- a/docker/runtime/doris-compose/command.py
+++ b/docker/runtime/doris-compose/command.py
@@ -31,8 +31,14 @@ LOG = utils.get_logger()
# return for_all, related_nodes, related_node_num
-def get_ids_related_nodes(cluster, fe_ids, be_ids, ignore_not_exists=False):
- if fe_ids is None and be_ids is None:
+def get_ids_related_nodes(cluster,
+ fe_ids,
+ be_ids,
+ ms_ids,
+ recycle_ids,
+ fdb_ids,
+ ignore_not_exists=False):
+ if fe_ids is None and be_ids is None and ms_ids is None and recycle_ids is None and fdb_ids is None:
return True, None, cluster.get_all_nodes_num()
def get_ids_related_nodes_with_type(node_type, ids):
@@ -55,9 +61,17 @@ def get_ids_related_nodes(cluster, fe_ids, be_ids, ignore_not_exists=False):
raise e
return nodes
- nodes = get_ids_related_nodes_with_type(
- CLUSTER.Node.TYPE_FE, fe_ids) + get_ids_related_nodes_with_type(
- CLUSTER.Node.TYPE_BE, be_ids)
+ type_ids = [
+ (CLUSTER.Node.TYPE_FE, fe_ids),
+ (CLUSTER.Node.TYPE_BE, be_ids),
+ (CLUSTER.Node.TYPE_MS, ms_ids),
+ (CLUSTER.Node.TYPE_RECYCLE, recycle_ids),
+ (CLUSTER.Node.TYPE_FDB, fdb_ids),
+ ]
+
+ nodes = []
+ for node_type, ids in type_ids:
+ nodes.extend(get_ids_related_nodes_with_type(node_type, ids))
related_node_num = len(nodes)
@@ -69,6 +83,9 @@ class Command(object):
def __init__(self, name):
self.name = name
+ def print_use_time(self):
+ return True
+
def add_parser(self, args_parsers):
raise Exception("No implemented")
@@ -88,13 +105,37 @@ class Command(object):
"if specific --fe-id but not specific ids, apply to all fe. Example: '--fe-id 2 3' will select fe-2 and fe-3.")
group.add_argument("--be-id", nargs="*", type=int, help="Specify up be ids, support multiple ids, " \
"if specific --be-id but not specific ids, apply to all be. Example: '--be-id' will select all backends.")
+ group.add_argument(
+ "--ms-id",
+ nargs="*",
+ type=int,
+ help=
+ "Specify up ms ids, support multiple ids. Only use in cloud cluster."
+ )
+ group.add_argument(
+ "--recycle-id",
+ nargs="*",
+ type=int,
+ help=
+ "Specify up recycle ids, support multiple ids. Only use in cloud cluster."
+ )
+ group.add_argument(
+ "--fdb-id",
+ nargs="*",
+ type=int,
+ help=
+ "Specify up fdb ids, support multiple ids. Only use in cloud cluster."
+ )
def _get_parser_bool_action(self, is_store_true):
- if sys.version_info.major == 3 and sys.version_info.minor >= 9:
+ if self._support_boolean_action():
return argparse.BooleanOptionalAction
else:
return "store_true" if is_store_true else "store_false"
+ def _support_boolean_action(self):
+ return sys.version_info.major == 3 and sys.version_info.minor >= 9
+
class SimpleCommand(Command):
@@ -104,7 +145,8 @@ class SimpleCommand(Command):
self.help = help
def add_parser(self, args_parsers):
- help = self.help + " If none of --fe-id, --be-id is specific, then apply to all containers."
+ help = self.help + " If none of --fe-id, --be-id, --ms-id, --recycle-id, --fdb-id is specific, "\
+ "then apply to all containers."
parser = args_parsers.add_parser(self.command, help=help)
parser.add_argument("NAME", help="Specify cluster name.")
self._add_parser_ids_args(parser)
@@ -113,7 +155,8 @@ class SimpleCommand(Command):
def run(self, args):
cluster = CLUSTER.Cluster.load(args.NAME)
_, related_nodes, related_node_num = get_ids_related_nodes(
- cluster, args.fe_id, args.be_id)
+ cluster, args.fe_id, args.be_id, args.ms_id, args.recycle_id,
+ args.fdb_id)
utils.exec_docker_compose_command(cluster.get_compose_file(),
self.command,
nodes=related_nodes)
@@ -128,7 +171,8 @@ class UpCommand(Command):
def add_parser(self, args_parsers):
parser = args_parsers.add_parser("up", help="Create and upgrade doris containers, "\
"or add new containers. " \
- "If none of --add-fe-num, --add-be-num, --fe-id, --be-id is specific, " \
+ "If none of --add-fe-num, --add-be-num, --add-ms-num, --add-recycle-num, "\
+ "--fe-id, --be-id, --ms-id, --recycle-id, --fdb-id is specific, " \
"then apply to all containers.")
parser.add_argument("NAME", default="", help="Specific cluster name.")
parser.add_argument("IMAGE",
@@ -136,6 +180,14 @@ class UpCommand(Command):
nargs="?",
help="Specify docker image.")
+ parser.add_argument(
+ "--cloud",
+ default=False,
+ action=self._get_parser_bool_action(True),
+ help=
+ "Create cloud cluster, default is false. Only use when creating new cluster."
+ )
+
parser.add_argument(
"--wait-timeout",
type=int,
@@ -161,6 +213,18 @@ class UpCommand(Command):
help=
"Specify add be num, default: 3 for a new cluster, 0 for a existing cluster."
)
+ group1.add_argument(
+ "--add-ms-num",
+ type=int,
+ help=
+ "Specify add ms num, default: 1 for a new cloud cluster, 0 for a existing cluster. Only use in cloud cluster"
+ )
+ group1.add_argument(
+ "--add-recycle-num",
+ type=int,
+ help=
+ "Specify add recycle num, default: 1 for a new cloud cluster, 0 for a existing cluster. Only use in cloud cluster"
+ )
group1.add_argument("--fe-config",
nargs="*",
type=str,
@@ -180,16 +244,32 @@ class UpCommand(Command):
"Example: --be-disks \"HDD=1\", \"SSD=1,10\", \"SSD=2,100\""\
"means each be has 1 HDD without capactity limit, 1 SSD with 10GB capactity limit, "\
"2 SSD with 100GB capactity limit")
+ group1.add_argument(
+ "--be-cluster",
+ type=str,
+ help=
+ "be cluster name, if not specific, will use compute_cluster. Only use in cloud cluster."
+ )
self._add_parser_ids_args(parser)
group2 = parser.add_mutually_exclusive_group()
- group2.add_argument(
- "--start",
- default=True,
- action=self._get_parser_bool_action(False),
- help="Start containers, default is true. If specific --no-start, "\
- "will create or update config image only but not start containers.")
+ if self._support_boolean_action():
+ group2.add_argument(
+ "--start",
+ default=True,
+ action=self._get_parser_bool_action(False),
+ help="Start containers, default is true. If specific --no-start, "\
+ "will create or update config image only but not start containers.")
+ else:
+ group2.add_argument(
+ "--no-start",
+ dest='start',
+ default=True,
+ action=self._get_parser_bool_action(False),
+ help=
+ "Create or update config image only and don't start containers."
+ )
group2.add_argument("--force-recreate",
default=False,
action=self._get_parser_bool_action(True),
@@ -200,13 +280,31 @@ class UpCommand(Command):
default="",
help="code coverage output directory")
+ parser.add_argument(
+ "--fdb-version",
+ type=str,
+ default="7.1.26",
+ help="fdb image version. Only use in cloud cluster.")
+
def run(self, args):
if not args.NAME:
raise Exception("Need specific not empty cluster name")
for_all = True
+ add_fdb_num = 0
try:
cluster = CLUSTER.Cluster.load(args.NAME)
- if args.fe_id != None or args.be_id != None or args.add_fe_num or args.add_be_num:
+
+ if not cluster.is_cloud:
+ args.add_ms_num = None
+ args.add_recycle_num = None
+ args.ms_id = None
+ args.recycle_id = None
+ args.fdb_id = None
+
+ if args.fe_id != None or args.be_id != None \
+ or args.ms_id != None or args.recycle_id != None or args.fdb_id != None \
+ or args.add_fe_num or args.add_be_num \
+ or args.add_ms_num or args.add_recycle_num:
for_all = False
except:
# a new cluster
@@ -220,37 +318,83 @@ class UpCommand(Command):
args.be_id = None
LOG.warning(
utils.render_yellow("Ignore --be-id for new cluster"))
- cluster = CLUSTER.Cluster.new(args.NAME, args.IMAGE,
+
+ args.fdb_id = None
+ args.ms_id = None
+ args.recycle_id = None
+
+ if args.add_fe_num is None:
+ args.add_fe_num = 3
+ if args.add_be_num is None:
+ args.add_be_num = 3
+
+ cloud_store_config = {}
+ if args.cloud:
+ add_fdb_num = 1
+ if not args.add_ms_num:
+ args.add_ms_num = 1
+ if not args.add_recycle_num:
+ args.add_recycle_num = 1
+ if not args.be_cluster:
+ args.be_cluster = "compute_cluster"
+ cloud_store_config = self._get_cloud_store_config()
+ else:
+ args.add_ms_num = 0
+ args.add_recycle_num = 0
+
+ cluster = CLUSTER.Cluster.new(args.NAME, args.IMAGE, args.cloud,
args.fe_config, args.be_config,
- args.be_disks, args.coverage_dir)
+ args.be_disks, args.be_cluster,
+ args.coverage_dir,
+ cloud_store_config)
LOG.info("Create new cluster {} succ, cluster path is {}".format(
args.NAME, cluster.get_path()))
- if not args.add_fe_num:
- args.add_fe_num = 3
- if not args.add_be_num:
- args.add_be_num = 3
+
+ if args.be_cluster and cluster.is_cloud:
+ cluster.be_cluster = args.be_cluster
_, related_nodes, _ = get_ids_related_nodes(cluster, args.fe_id,
- args.be_id)
- add_be_ids = []
+ args.be_id, args.ms_id,
+ args.recycle_id,
+ args.fdb_id)
add_fe_ids = []
+ add_be_ids = []
+ add_ms_ids = []
+ add_recycle_ids = []
+ add_fdb_ids = []
+
+ add_type_nums = [
+ (CLUSTER.Node.TYPE_FDB, add_fdb_num, add_fdb_ids),
+ (CLUSTER.Node.TYPE_MS, args.add_ms_num, add_ms_ids),
+ (CLUSTER.Node.TYPE_RECYCLE, args.add_recycle_num, add_recycle_ids),
+ (CLUSTER.Node.TYPE_FE, args.add_fe_num, add_fe_ids),
+ (CLUSTER.Node.TYPE_BE, args.add_be_num, add_be_ids),
+ ]
+
if not related_nodes:
related_nodes = []
- if args.add_fe_num:
- for i in range(args.add_fe_num):
- fe = cluster.add(CLUSTER.Node.TYPE_FE)
- related_nodes.append(fe)
- add_fe_ids.append(fe.id)
- if args.add_be_num:
- for i in range(args.add_be_num):
- be = cluster.add(CLUSTER.Node.TYPE_BE)
- related_nodes.append(be)
- add_be_ids.append(be.id)
+
+ def do_add_node(node_type, add_num, add_ids):
+ if not add_num:
+ return
+ for i in range(add_num):
+ node = cluster.add(node_type)
+ related_nodes.append(node)
+ add_ids.append(node.id)
+
+ for node_type, add_num, add_ids in add_type_nums:
+ do_add_node(node_type, add_num, add_ids)
+
if args.IMAGE:
for node in related_nodes:
node.set_image(args.IMAGE)
- if for_all and args.IMAGE:
- cluster.set_image(args.IMAGE)
+ if for_all:
+ cluster.set_image(args.IMAGE)
+
+ for node in cluster.get_all_nodes(CLUSTER.Node.TYPE_FDB):
+ node.set_image("foundationdb/foundationdb:{}".format(
+ args.fdb_version))
+
cluster.save()
options = []
@@ -269,6 +413,8 @@ class UpCommand(Command):
utils.exec_docker_compose_command(cluster.get_compose_file(), "up",
options, related_nodes)
+ ls_cmd = "python docker/runtime/doris-compose/doris-compose.py ls " + cluster.name
+ LOG.info("Inspect command: " + utils.render_green(ls_cmd) + "\n")
LOG.info(
"Master fe query address: " +
utils.render_green(CLUSTER.get_master_fe_endpoint(cluster.name)) +
@@ -318,8 +464,59 @@ class UpCommand(Command):
"be": {
"add_list": add_be_ids,
},
+ "ms": {
+ "add_list": add_ms_ids,
+ },
+ "recycle": {
+ "add_list": add_recycle_ids,
+ },
+ "fdb": {
+ "add_list": add_fdb_ids,
+ },
}
+ def _get_cloud_store_config(self):
+ example_cfg_file = os.path.join(CLUSTER.LOCAL_RESOURCE_PATH,
+ "cloud.ini.example")
+ if not CLUSTER.CLOUD_CFG_FILE:
+ raise Exception("Cloud cluster need S3 store, specific its config in a file.\n" \
+ "A example file is " + example_cfg_file + ".\n" \
+ "Then setting the env variable `export DORIS_CLOUD_CFG_FILE=<cfg-file-path>`.")
+
+ if not os.path.exists(CLUSTER.CLOUD_CFG_FILE):
+ raise Exception("Cloud store config file '" +
+ CLUSTER.CLOUD_CFG_FILE + "' not exists.")
+
+ config = {}
+ with open(example_cfg_file, "r") as f:
+ for line in f.readlines():
+ if line.startswith('#'):
+ continue
+ pos = line.find('=')
+ if pos <= 0:
+ continue
+ key = line[0:pos].strip()
+ if key:
+ config[key] = ""
+
+ with open(CLUSTER.CLOUD_CFG_FILE, "r") as f:
+ for line in f.readlines():
+ if line.startswith('#'):
+ continue
+ pos = line.find('=')
+ if pos <= 0:
+ continue
+ key = line[0:pos].strip()
+ if key and config.get(key, None) != None:
+ config[key] = line[line.find('=') + 1:].strip()
+
+ for key, value in config.items():
+ if not value:
+ raise Exception(
+ "Should provide none empty property '{}' in file {}".
+ format(key, CLUSTER.CLOUD_CFG_FILE))
+ return config
+
class DownCommand(Command):
@@ -327,7 +524,7 @@ class DownCommand(Command):
parser = args_parsers.add_parser("down",
help="Down doris containers, networks. "\
"It will also remove node from DB. " \
- "If none of --fe-id, --be-id is specific, "\
+ "If none of --fe-id, --be-id, --ms-id, --recycle-id, --fdb-id is specific, "\
"then apply to all containers.")
parser.add_argument("NAME", help="Specify cluster name")
self._add_parser_ids_args(parser)
@@ -353,7 +550,13 @@ class DownCommand(Command):
except:
return "Cluster not exists or load failed"
for_all, related_nodes, related_node_num = get_ids_related_nodes(
- cluster, args.fe_id, args.be_id, ignore_not_exists=True)
+ cluster,
+ args.fe_id,
+ args.be_id,
+ args.ms_id,
+ args.recycle_id,
+ args.fdb_id,
+ ignore_not_exists=True)
if for_all:
if os.path.exists(cluster.get_compose_file()):
@@ -401,8 +604,9 @@ class DownCommand(Command):
if args.clean:
utils.enable_dir_with_rw_perm(node.get_path())
shutil.rmtree(node.get_path())
- register_file = "{}/{}-register".format(
- CLUSTER.get_status_path(cluster.name), node.get_ip())
+ register_file = "{}/{}-{}-register".format(
+ CLUSTER.get_status_path(cluster.name),
+ node.node_type(), node.get_ip())
if os.path.exists(register_file):
os.remove(register_file)
LOG.info(
@@ -444,8 +648,8 @@ class ListNode(object):
result = [
self.cluster_name, "{}-{}".format(self.node_type, self.id),
self.ip, self.status, self.container_id, self.image, self.created,
- self.alive, self.is_master, self.query_port, self.backend_id,
- self.tablet_num, self.last_heartbeat, self.err_msg
+ self.alive, self.is_master, self.backend_id, self.tablet_num,
+ self.last_heartbeat, self.err_msg
]
if detail:
query_port = ""
@@ -483,6 +687,48 @@ class ListNode(object):
self.err_msg = be.err_msg
+class GenConfCommand(Command):
+
+ def print_use_time(self):
+ return False
+
+ def add_parser(self, args_parsers):
+ parser = args_parsers.add_parser(
+ "config",
+ help="Generate regression-conf-custom.groovy for regression test.")
+ parser.add_argument("NAME", default="", help="Specific cluster name.")
+
+ return parser
+
+ def run(self, args):
+ content = '''
+jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true"
+targetJdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true"
+feSourceThriftAddress = "127.0.0.1:9020"
+feTargetThriftAddress = "127.0.0.1:9020"
+syncerAddress = "127.0.0.1:9190"
+feHttpAddress = "127.0.0.1:8030"
+'''
+ master_fe_ip = CLUSTER.get_master_fe_endpoint(args.NAME)
+ if not master_fe_ip:
+ print("Not found cluster with name {} in directory {}".format(
+ args.NAME, CLUSTER.LOCAL_DORIS_PATH))
+ return
+ doris_root_dir = os.path.abspath(__file__)
+ for i in range(4):
+ doris_root_dir = os.path.dirname(doris_root_dir)
+ regression_conf_custom = doris_root_dir + "/regression-test/conf/regression-conf-custom.groovy"
+ if input("write file {} ?\n y/N: ".format(
+ regression_conf_custom)) != 'y':
+ print("No write regression custom file.")
+ return
+ with open(regression_conf_custom, "w") as f:
+ f.write(
+ content.replace("127.0.0.1",
+ master_fe_ip[:master_fe_ip.find(':')]))
+ print("Write succ: " + regression_conf_custom)
+
+
class ListCommand(Command):
def add_parser(self, args_parsers):
@@ -574,7 +820,7 @@ class ListCommand(Command):
TYPE_COMPOSESERVICE = type(ComposeService("", "", ""))
if not args.NAME:
- header = ("CLUSTER", "OWNER", "STATUS", "MASTER FE",
+ header = ("CLUSTER", "OWNER", "STATUS", "MASTER FE", "CLOUD",
"CONFIG FILES")
rows = []
for name in sorted(clusters.keys()):
@@ -591,16 +837,24 @@ class ListCommand(Command):
])
owner = utils.get_path_owner(CLUSTER.get_cluster_path(name))
compose_file = CLUSTER.get_compose_file(name)
+
+ is_cloud = ""
+ try:
+ cluster = CLUSTER.Cluster.load(name)
+ is_cloud = "true" if cluster.is_cloud else "false"
+ except:
+ pass
+
rows.append((name, owner, show_status,
- CLUSTER.get_master_fe_endpoint(name),
+ CLUSTER.get_master_fe_endpoint(name), is_cloud,
"{}{}".format(compose_file,
cluster_info["status"])))
return self._handle_data(header, rows)
header = [
"CLUSTER", "NAME", "IP", "STATUS", "CONTAINER ID", "IMAGE",
- "CREATED", "alive", "is_master", "query_port", "backend_id",
- "tablet_num", "last_heartbeat", "err_msg"
+ "CREATED", "alive", "is_master", "backend_id", "tablet_num",
+ "last_heartbeat", "err_msg"
]
if args.detail:
header += [
@@ -644,6 +898,11 @@ class ListCommand(Command):
node.image = ",".join(container.image.tags)
node.container_id = container.short_id
node.status = container.status
+ if node.container_id and \
+ node_type in (CLUSTER.Node.TYPE_FDB,
+ CLUSTER.Node.TYPE_MS,
+ CLUSTER.Node.TYPE_RECYCLE):
+ node.alive = "true"
for id, fe in db_mgr.fe_states.items():
if fe_ids.get(id, False):
@@ -666,17 +925,10 @@ class ListCommand(Command):
node.update_db_info(db_mgr)
nodes.append(node)
- def get_key(node):
- key = node.id
- if node.node_type == CLUSTER.Node.TYPE_FE:
- key += 0 * CLUSTER.ID_LIMIT
- elif node.node_type == CLUSTER.Node.TYPE_BE:
- key += 1 * CLUSTER.ID_LIMIT
- else:
- key += 2 * CLUSTER.ID_LIMIT
- return key
+ def get_node_seq(node):
+ return CLUSTER.get_node_seq(node.node_type, node.id)
- for node in sorted(nodes, key=get_key):
+ for node in sorted(nodes, key=get_node_seq):
rows.append(node.info(args.detail))
return self._handle_data(header, rows)
@@ -690,5 +942,6 @@ ALL_COMMANDS = [
SimpleCommand("restart", "Restart the doris containers. "),
SimpleCommand("pause", "Pause the doris containers. "),
SimpleCommand("unpause", "Unpause the doris containers. "),
+ GenConfCommand("config"),
ListCommand("ls"),
]
diff --git a/docker/runtime/doris-compose/doris-compose.py b/docker/runtime/doris-compose/doris-compose.py
index c29449b116d..0091b70eae9 100644
--- a/docker/runtime/doris-compose/doris-compose.py
+++ b/docker/runtime/doris-compose/doris-compose.py
@@ -36,7 +36,7 @@ def run(args, disable_log, help):
if args.command == cmd.name:
timer = utils.Timer()
result = cmd.run(args)
- if not disable_log:
+ if cmd.print_use_time() and not disable_log:
timer.show()
return result
print(help)
diff --git a/docker/runtime/doris-compose/resource/cloud.ini.example b/docker/runtime/doris-compose/resource/cloud.ini.example
new file mode 100644
index 00000000000..2ce113a2b53
--- /dev/null
+++ b/docker/runtime/doris-compose/resource/cloud.ini.example
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+DORIS_CLOUD_USER=xxxx
+DORIS_CLOUD_AK=xxxx
+DORIS_CLOUD_SK=xxxx
+DORIS_CLOUD_BUCKET=xxxx
+DORIS_CLOUD_ENDPOINT=cos.ap-hongkong.myqcloud.com
+DORIS_CLOUD_EXTERNAL_ENDPOINT=cos.ap-hongkong.myqcloud.com
+DORIS_CLOUD_REGION=ap-hongkong
+DORIS_CLOUD_PROVIDER=COS
diff --git a/docker/runtime/doris-compose/resource/common.sh b/docker/runtime/doris-compose/resource/common.sh
index 5342eb0e54b..de6ba29865a 100644
--- a/docker/runtime/doris-compose/resource/common.sh
+++ b/docker/runtime/doris-compose/resource/common.sh
@@ -17,21 +17,102 @@
export MASTER_FE_IP=""
export MASTER_FE_IP_FILE=$DORIS_HOME/status/master_fe_ip
+export HAS_INIT_FDB_FILE=${DORIS_HOME}/status/has_init_fdb
+export HAS_CREATE_INSTANCE_FILE=$DORIS_HOME/status/has_create_instance
export LOG_FILE=$DORIS_HOME/log/health.out
+export LOCK_FILE=$DORIS_HOME/status/token
health_log() {
echo "$(date +'%Y-%m-%d %H:%M:%S') $@" >>$LOG_FILE
}
-read_master_fe_ip() {
- MASTER_FE_IP=$(cat $MASTER_FE_IP_FILE)
- if [ $? -eq 0 ]; then
- health_log "master fe ${MASTER_FE_IP} has ready."
- return 0
- else
+# concurrent write meta service server will failed due to fdb txn conflict.
+# so add lock to protect writing ms txns.
+lock_cluster() {
+ health_log "start acquire token"
+ while true; do
+ if [ -f $LOCK_FILE ]; then
+ if [ "a$(cat $LOCK_FILE)" == "a${MY_IP}" ]; then
+ health_log "rm $LOCK_FILE generate by myself"
+ rm $LOCK_FILE
+ continue
+ fi
+
+ mt=$(stat -c %Y $LOCK_FILE)
+ if [ -z "$mt" ]; then
+ health_log "get $LOCK_FILE modify time failed"
+ sleep 0.1
+ continue
+ fi
+
+ now=$(date '+%s')
+ diff=$(expr $now - $mt)
+ if [ $diff -lt 10 ]; then
+ sleep 0.1
+ continue
+ fi
+
+ rm $LOCK_FILE
+ health_log "rm $LOCK_FILE due to exceeds $diff seconds."
+ fi
+
+ if [ ! -f $LOCK_FILE ]; then
+ echo $MY_IP >$LOCK_FILE
+ fi
+
+ sleep 0.1
+
+ if [ "a$(cat $LOCK_FILE)" == "a${MY_IP}" ]; then
+ break
+ fi
+
+ sleep 0.1
+ done
+
+ health_log "now got token"
+}
+
+unlock_cluster() {
+ if [ ! -f $LOCK_FILE ]; then
+ return
+ fi
+
+ if [ "a$(cat $LOCK_FILE)" == "a${MY_IP}" ]; then
+ rm $LOCK_FILE
+ fi
+}
+
+wait_master_fe_ready() {
+ while true; do
+ MASTER_FE_IP=$(cat $MASTER_FE_IP_FILE)
+ if [ -n "$MASTER_FE_IP" ]; then
+ health_log "master fe ${MASTER_FE_IP} has ready."
+ break
+ fi
health_log "master fe has not ready."
- return 1
+ sleep 1
+ done
+}
+
+wait_create_instance() {
+ ok=0
+ for ((i = 0; i < 30; i++)); do
+ if [ -f $HAS_CREATE_INSTANCE_FILE ]; then
+ ok=1
+ break
+ fi
+
+ health_log "has not create instance, not found file $HAS_CREATE_INSTANCE_FILE"
+
+ sleep 1
+ done
+
+ if [ $ok -eq 0 ]; then
+ health_log "wait create instance file too long, exit"
+ exit 1
fi
+
+ health_log "check has create instance ok"
}
wait_pid() {
diff --git a/docker/runtime/doris-compose/resource/fdb.conf b/docker/runtime/doris-compose/resource/fdb.conf
new file mode 100644
index 00000000000..14c9976e4fc
--- /dev/null
+++ b/docker/runtime/doris-compose/resource/fdb.conf
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[fdbmonitor]
+user = root
+group = root
+
+[general]
+restart-delay = 60
+cluster-file = /opt/apache-doris/fdb/conf/fdb.cluster
+
+## Default parameters for individual fdbserver processes
+[fdbserver]
+command = /usr/bin/fdbserver
+public-address = ${PUBLISH-ADDRESS}
+listen-address = public
+logdir = /opt/apache-doris/fdb/log
+datadir = /opt/apache-doris/fdb/data
+logsize = 10MiB
+maxlogssize = 100MiB
+memory = 1GiB
+storage-memory = 200MiB
+cache-memory = 200MiB
+
+## An individual fdbserver process with id 4500
+## Parameters set here override defaults from the [fdbserver] section
+[fdbserver.4500]
+
+[backup_agent]
+command = /usr/bin/backup_agent
+logdir = /opt/apache-doris/fdb/log
diff --git a/docker/runtime/doris-compose/resource/init_be.sh b/docker/runtime/doris-compose/resource/init_be.sh
index 4d7b9780c30..0df464c625c 100755
--- a/docker/runtime/doris-compose/resource/init_be.sh
+++ b/docker/runtime/doris-compose/resource/init_be.sh
@@ -22,20 +22,17 @@ DIR=$(
source $DIR/common.sh
-REGISTER_FILE=$DORIS_HOME/status/$MY_IP-register
+REGISTER_FILE=$DORIS_HOME/status/be-$MY_IP-register
+
+add_local_be() {
+ wait_master_fe_ready
-add_backend() {
while true; do
- read_master_fe_ip
- if [ $? -ne 0 ]; then
- sleep 1
- continue
- fi
- lsof -i:$BE_HEARTBEAT_PORT
- if [ $? -ne 0 ]; then
- sleep 1
- continue
- fi
+ #lsof -i:$BE_HEARTBEAT_PORT
+ #if [ $? -ne 0 ]; then
+ # sleep 1
+ # continue
+ #fi
output=$(mysql -P $FE_QUERY_PORT -h $MASTER_FE_IP -u root --execute "ALTER SYSTEM ADD BACKEND '$MY_IP:$BE_HEARTBEAT_PORT';" 2>&1)
res=$?
@@ -44,8 +41,80 @@ add_backend() {
(echo $output | grep "Same backend already exists") && break
sleep 1
done
+}
- touch $REGISTER_FILE
+add_cloud_be() {
+ if [ -f "${DORIS_HOME}/log/be.out" ]; then
+ return
+ fi
+
+ cluster_file_name="${DORIS_HOME}/conf/CLUSTER_NAME"
+ cluster_name=$(cat $cluster_file_name)
+ if [ -z $cluster_name ]; then
+ health_log "Empty cluster name, it should specific in file ${cluster_file_name}"
+ exit 1
+ fi
+
+ cluster_id="${cluster_name}_id"
+
+ wait_create_instance
+
+ nodes='{
+ "cloud_unique_id": "'"${CLOUD_UNIQUE_ID}"'",
+ "ip": "'"${MY_IP}"'",
+ "heartbeat_port": "'"${BE_HEARTBEAT_PORT}"'"
+ }'
+
+ lock_cluster
+
+ output=$(curl -s "${META_SERVICE_ENDPOINT}/MetaService/http/add_cluster?token=greedisgood9999" \
+ -d '{"instance_id": "default_instance_id",
+ "cluster": {
+ "type": "COMPUTE",
+ "cluster_name": "'"${cluster_name}"'",
+ "cluster_id": "'"${cluster_id}"'",
+ "nodes": ['"${nodes}"']
+ }}')
+
+ health_log "add cluster. output: $output"
+ code=$(jq -r '.code' <<<$output)
+
+ # cluster has exists
+ if [ "$code" == "ALREADY_EXISTED" ]; then
+ output=$(curl -s "${META_SERVICE_ENDPOINT}/MetaService/http/add_node?token=greedisgood9999" \
+ -d '{"instance_id": "default_instance_id",
+ "cluster": {
+ "type": "COMPUTE",
+ "cluster_name": "'"${cluster_name}"'",
+ "cluster_id": "'"${cluster_id}"'",
+ "nodes": ['"${nodes}"']
+ }}')
+ fi
+
+ unlock_cluster
+
+ health_log "add cluster. output: $output"
+ code=$(jq -r '.code' <<<$output)
+
+ if [ "$code" != "OK" ]; then
+ health_log "add cluster failed, exit."
+ exit 1
+ fi
+
+ output=$(curl -s "${META_SERVICE_ENDPOINT}/MetaService/http/get_cluster?token=greedisgood9999" \
+ -d '{"instance_id": "default_instance_id",
+ "cloud_unique_id": "'"${CLOUD_UNIQUE_ID}"'",
+ "cluster_name": "'"${cluster_name}"'",
+ "cluster_id": "'"${cluster_id}"'"
+ }')
+
+ health_log "get cluster is: $output"
+ code=$(jq -r '.code' <<<$output)
+
+ if [ "$code" != "OK" ]; then
+ health_log "get cluster failed, exit."
+ exit 1
+ fi
}
stop_backend() {
@@ -63,7 +132,7 @@ wait_process() {
for ((i = 0; i < 5; i++)); do
sleep 1s
pid=$(pgrep doris_be)
- if [ -n $pid ]; then
+ if [ -n "$pid" ]; then
break
fi
done
@@ -71,16 +140,30 @@ wait_process() {
wait_pid $pid
}
-main() {
- trap stop_backend SIGTERM
+add_be_to_cluster() {
+ if [ -f $REGISTER_FILE ]; then
+ return
+ fi
- if [ ! -f $REGISTER_FILE ]; then
- add_backend &
+ if [ "${IS_CLOUD}" == "1" ]; then
+ add_cloud_be
+ else
+ add_local_be
fi
- if [ -n $LLVM_PROFILE_FILE_PREFIX ]; then
+ touch $REGISTER_FILE
+ health_log "register be"
+}
+
+main() {
+ trap stop_backend SIGTERM
+
+ if [ -n "$LLVM_PROFILE_FILE_PREFIX" ]; then
export LLVM_PROFILE_FILE="${LLVM_PROFILE_FILE_PREFIX}-$(date +%s)"
fi
+
+ add_be_to_cluster
+
health_log "run start_be.sh"
bash $DORIS_HOME/bin/start_be.sh --daemon
diff --git a/docker/runtime/doris-compose/resource/init_cloud.sh b/docker/runtime/doris-compose/resource/init_cloud.sh
new file mode 100644
index 00000000000..baa983aee95
--- /dev/null
+++ b/docker/runtime/doris-compose/resource/init_cloud.sh
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARGS=$@
+
+DIR=$(
+ cd $(dirname $0)
+ pwd
+)
+
+source $DIR/common.sh
+
+wait_fdb_ready() {
+ while true; do
+ if [ -f $HAS_INIT_FDB_FILE ]; then
+ health_log "has init fdb"
+ return
+ fi
+
+ health_log "hasn't init fdb, waiting"
+
+ sleep 1
+ done
+}
+
+check_init_cloud() {
+ if [ -f $HAS_CREATE_INSTANCE_FILE ]; then
+ return
+ fi
+
+ if [ "$MY_TYPE" != "ms" -o "$MY_ID" != "1" ]; then
+ return
+ fi
+
+ while true; do
+
+ lock_cluster
+
+ output=$(curl -s "${META_SERVICE_ENDPOINT}/MetaService/http/create_instance?token=greedisgood9999" \
+ -d '{"instance_id":"default_instance_id",
+ "name": "default_instance",
+ "user_id": "'"${DORIS_CLOUD_USER}"'",
+ "obj_info": {
+ "ak": "'"${DORIS_CLOUD_AK}"'",
+ "sk": "'"${DORIS_CLOUD_SK}"'",
+ "bucket": "'"${DORIS_CLOUD_BUCKET}"'",
+ "endpoint": "'"${DORIS_CLOUD_ENDPOINT}"'",
+ "external_endpoint": "'"${DORIS_CLOUD_EXTERNAL_ENDPOINT}"'",
+ "prefix": "'"${DORIS_CLOUD_PREFIX}"'",
+ "region": "'"${DORIS_CLOUD_REGION}"'",
+ "provider": "'"${DORIS_CLOUD_PROVIDER}"'"
+ }}')
+
+ unlock_cluster
+
+ health_log "create instance output: $output"
+ code=$(jq -r '.code' <<<$output)
+
+ if [ "$code" != "OK" ]; then
+ health_log "create instance failed"
+ sleep 1
+ continue
+ fi
+
+ health_log "create doris instance succ, output: $output"
+ touch $HAS_CREATE_INSTANCE_FILE
+ break
+ done
+}
+
+stop_cloud() {
+ cd $DORIS_HOME
+ bash bin/stop.sh
+}
+
+main() {
+ trap stop_cloud SIGTERM
+
+ cd $DORIS_HOME
+
+ wait_fdb_ready
+
+ check_init_cloud &
+
+ health_log "input args: $ARGS"
+
+ bash bin/start.sh $ARGS
+}
+
+main
diff --git a/docker/runtime/doris-compose/resource/common.sh b/docker/runtime/doris-compose/resource/init_fdb.sh
similarity index 54%
copy from docker/runtime/doris-compose/resource/common.sh
copy to docker/runtime/doris-compose/resource/init_fdb.sh
index 5342eb0e54b..b3b22f44ee4 100644
--- a/docker/runtime/doris-compose/resource/common.sh
+++ b/docker/runtime/doris-compose/resource/init_fdb.sh
@@ -15,41 +15,46 @@
# specific language governing permissions and limitations
# under the License.
-export MASTER_FE_IP=""
-export MASTER_FE_IP_FILE=$DORIS_HOME/status/master_fe_ip
-export LOG_FILE=$DORIS_HOME/log/health.out
+DIR=$(
+ cd $(dirname $0)
+ pwd
+)
-health_log() {
- echo "$(date +'%Y-%m-%d %H:%M:%S') $@" >>$LOG_FILE
-}
-
-read_master_fe_ip() {
- MASTER_FE_IP=$(cat $MASTER_FE_IP_FILE)
- if [ $? -eq 0 ]; then
- health_log "master fe ${MASTER_FE_IP} has ready."
- return 0
- else
- health_log "master fe has not ready."
- return 1
- fi
-}
+source $DIR/common.sh
-wait_pid() {
- pid=$1
- health_log ""
- health_log "ps -elf\n$(ps -elf)\n"
- if [ -z $pid ]; then
- health_log "pid not exist"
- exit 1
+init_db() {
+ if [ -f $HAS_INIT_FDB_FILE ]; then
+ return
fi
- health_log "wait process $pid"
- while true; do
- ps -p $pid >/dev/null
- if [ $? -ne 0 ]; then
+ for ((i = 0; i < 10; i++)); do
+ /usr/bin/fdbcli -C ${DORIS_HOME}/conf/fdb.cluster --exec 'configure new single ssd'
+ if [ $? -eq 0 ]; then
+ touch $HAS_INIT_FDB_FILE
+ health_log "fdbcli init cluster succ"
break
fi
- sleep 1s
+
+ health_log "fdbcli init cluster failed"
+ sleep 1
done
- health_log "wait end"
+
+ if [ ! -f $HAS_INIT_FDB_FILE ]; then
+ health_log "fdbcli init cluster failed, exit"
+ exit 1
+ fi
+}
+
+stop_fdb() {
+ exit 0
+}
+
+main() {
+ trap stop_fdb SIGTERM
+
+ init_db &
+
+ fdbmonitor --conffile ${DORIS_HOME}/conf/fdb.conf --lockfile ${DORIS_HOME}/fdbmonitor.pid
}
+
+main
diff --git a/docker/runtime/doris-compose/resource/init_fe.sh b/docker/runtime/doris-compose/resource/init_fe.sh
index 1762f10d1fa..e532d0d56e1 100755
--- a/docker/runtime/doris-compose/resource/init_fe.sh
+++ b/docker/runtime/doris-compose/resource/init_fe.sh
@@ -22,14 +22,12 @@ DIR=$(
source $DIR/common.sh
-add_frontend() {
- while true; do
- read_master_fe_ip
- if [ $? -ne 0 ]; then
- sleep 1
- continue
- fi
+REGISTER_FILE=$DORIS_HOME/status/fe-$MY_IP-register
+
+add_local_fe() {
+ wait_master_fe_ready
+ while true; do
output=$(mysql -P $FE_QUERY_PORT -h $MASTER_FE_IP -u root --execute "ALTER SYSTEM ADD FOLLOWER '$MY_IP:$FE_EDITLOG_PORT';" 2>&1)
res=$?
health_log "${output}\n"
@@ -37,6 +35,8 @@ add_frontend() {
(echo $output | grep "frontend already exists") && break
sleep 1
done
+
+ touch $REGISTER_FILE
}
fe_daemon() {
@@ -81,6 +81,69 @@ fe_daemon() {
done
}
+add_cloud_fe() {
+ if [ -f "$REGISTER_FILE" ]; then
+ return
+ fi
+
+ wait_create_instance
+
+ action=add_cluster
+ node_type=FE_MASTER
+ if [ "$MY_ID" != "1" ]; then
+ wait_master_fe_ready
+ action=add_node
+ node_type=FE_OBSERVER
+ fi
+
+ nodes='{
+ "cloud_unique_id": "'"${CLOUD_UNIQUE_ID}"'",
+ "ip": "'"${MY_IP}"'",
+ "edit_log_port": "'"${FE_EDITLOG_PORT}"'",
+ "node_type": "'"${node_type}"'"
+ }'
+
+ lock_cluster
+
+ output=$(curl -s "${META_SERVICE_ENDPOINT}/MetaService/http/${action}?token=greedisgood9999" \
+ -d '{"instance_id": "default_instance_id",
+ "cluster": {
+ "type": "SQL",
+ "cluster_name": "RESERVED_CLUSTER_NAME_FOR_SQL_SERVER",
+ "cluster_id": "RESERVED_CLUSTER_ID_FOR_SQL_SERVER",
+ "nodes": ['"${nodes}"']
+ }}')
+
+ unlock_cluster
+
+ health_log "add cluster. output: $output"
+ code=$(jq -r '.code' <<<$output)
+
+ if [ "$code" != "OK" ]; then
+ health_log "add cluster failed, exit."
+ exit 1
+ fi
+
+ output=$(curl -s "${META_SERVICE_ENDPOINT}/MetaService/http/get_cluster?token=greedisgood9999" \
+ -d '{"instance_id": "default_instance_id",
+ "cloud_unique_id": "'"${CLOUD_UNIQUE_ID}"'",
+ "cluster_name": "RESERVED_CLUSTER_NAME_FOR_SQL_SERVER",
+ "cluster_id": "RESERVED_CLUSTER_ID_FOR_SQL_SERVER"}')
+
+ health_log "get cluster is: $output"
+ code=$(jq -r '.code' <<<$output)
+
+ if [ "$code" != "OK" ]; then
+ health_log "get cluster failed, exit."
+ exit 1
+ fi
+
+ touch $REGISTER_FILE
+ if [ "$MY_ID" == "1" ]; then
+ echo $MY_IP >$MASTER_FE_IP_FILE
+ fi
+}
+
stop_frontend() {
if [ "$STOP_GRACE" = "1" ]; then
bash $DORIS_HOME/bin/stop_fe.sh --grace
@@ -96,7 +159,7 @@ wait_process() {
for ((i = 0; i < 5; i++)); do
sleep 1s
pid=$(ps -elf | grep java | grep org.apache.doris.DorisFE | grep -v grep | awk '{print $4}')
- if [ -n $pid ]; then
+ if [ -n "$pid" ]; then
break
fi
done
@@ -104,17 +167,34 @@ wait_process() {
wait_pid $pid
}
-main() {
- trap stop_frontend SIGTERM
+start_local_fe() {
+ if [ "$MY_ID" = "1" -a ! -f $REGISTER_FILE ]; then
+ touch $REGISTER_FILE
+ fi
- if [ "$MY_ID" = "1" -o -d "${DORIS_HOME}/doris-meta/image" ]; then
+ if [ -f $REGISTER_FILE ]; then
fe_daemon &
bash $DORIS_HOME/bin/start_fe.sh --daemon
else
- add_frontend
+ add_local_fe
fe_daemon &
bash $DORIS_HOME/bin/start_fe.sh --helper $MASTER_FE_IP:$FE_EDITLOG_PORT --daemon
fi
+}
+
+start_cloud_fe() {
+ add_cloud_fe
+ bash $DORIS_HOME/bin/start_fe.sh --daemon
+}
+
+main() {
+ trap stop_frontend SIGTERM
+
+ if [ "$IS_CLOUD" == "1" ]; then
+ start_cloud_fe
+ else
+ start_local_fe
+ fi
wait_process
}
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
index 27632326a3b..a83722be2ea 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
@@ -216,9 +216,20 @@ class Suite implements GroovyInterceptable {
return
}
+ boolean pipelineIsCloud = isCloudCluster()
+ boolean dockerIsCloud = false
+ if (options.cloudMode == null) {
+ dockerIsCloud = pipelineIsCloud
+ } else {
+ dockerIsCloud = options.cloudMode
+ if (dockerIsCloud != pipelineIsCloud && options.skipRunWhenPipelineDiff) {
+ return
+ }
+ }
+
try {
cluster.destroy(true)
- cluster.init(options)
+ cluster.init(options, dockerIsCloud)
def user = context.config.jdbcUser
def password = context.config.jdbcPassword
@@ -971,6 +982,10 @@ class Suite implements GroovyInterceptable {
return result.last().get(0);
}
+ boolean isCloudCluster() {
+ return !getFeConfig("cloud_unique_id").isEmpty()
+ }
+
String getFeConfig(String key) {
return sql_return_maparray("SHOW FRONTEND CONFIG LIKE '${key}'")[0].Value
}
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteCluster.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteCluster.groovy
index 6d4ae4be0ad..e556c1896bb 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteCluster.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteCluster.groovy
@@ -38,6 +38,15 @@ class ClusterOptions {
List<String> beConfigs = []
boolean connectToFollower = false
+ // 1. cloudMode = true, only create cloud cluster.
+ // 2. cloudMode = false, only create none-cloud cluster.
+ // 3. cloudMode = null, create both cloud and none-cloud cluster, depend on the running pipeline mode.
+ Boolean cloudMode = false
+
+ // when cloudMode = true/false, but the running pipeline is diff with cloudMode,
+ // skip run this docker test or not.
+ boolean skipRunWhenPipelineDiff = true
+
// each be disks, a disks format is: disk_type=disk_num[,disk_capacity]
// here disk_type=HDD or SSD, disk capacity is in gb unit.
// for example: beDisks = ["HDD=1", "SSD=2,10", "SSD=10,3"] means:
@@ -169,7 +178,7 @@ class SuiteCluster {
this.running = false
}
- void init(ClusterOptions options) {
+ void init(ClusterOptions options, boolean isCloud) {
assert name != null && name != ''
assert options.feNum > 0 || options.beNum > 0
assert config.image != null && config.image != ''
@@ -200,6 +209,9 @@ class SuiteCluster {
if (config.dockerCoverageOutputDir != null && config.dockerCoverageOutputDir != '') {
cmd += ['--coverage-dir', config.dockerCoverageOutputDir]
}
+ if (isCloud) {
+ cmd += ['--cloud']
+ }
cmd += ['--wait-timeout', String.valueOf(180)]
runCmd(cmd.join(' '), -1)
@@ -296,6 +308,8 @@ class SuiteCluster {
} else if (name.startsWith('fe-')) {
int index = name.substring('fe-'.length()) as int
frontends.add(Frontend.fromCompose(header, index, row))
+ } else if (name.startsWith('ms-') || name.startsWith('recycle-') || name.startsWith('fdb-')) {
+ // TODO: handle these nodes
} else {
assert false : 'Unknown node type with name: ' + name
}
diff --git a/regression-test/suites/demo_p0/docker_action.groovy b/regression-test/suites/demo_p0/docker_action.groovy
index 8c827aa40f4..6d62d6ea7be 100644
--- a/regression-test/suites/demo_p0/docker_action.groovy
+++ b/regression-test/suites/demo_p0/docker_action.groovy
@@ -50,4 +50,19 @@ suite('docker_action') {
docker(options) {
sql '''create table tb1 (k int) DISTRIBUTED BY HASH(k) BUCKETS 10 properties ("replication_num"="5")'''
}
+
+ def options2 = new ClusterOptions()
+ options2.beNum = 1
+ // create cloud cluster
+ options2.cloudMode = true
+ //// cloud docker only run in cloud pipeline, but enable it run in none-cloud pipeline
+ // options2.skipRunWhenPipelineDiff = false
+ // run another docker, create a cloud cluster
+ docker(options2) {
+ // cloud cluster will ignore replication_num, always set to 1. so create table succ even has 1 be.
+ sql '''create table tb1 (k int) DISTRIBUTED BY HASH(k) BUCKETS 10 properties ("replication_num"="1000")'''
+
+ sql 'set global enable_memtable_on_sink_node = false'
+ sql 'insert into tb1 values (2)'
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/06: Fix cgroup cpu controller file leak (#31398)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 602f05b5215ff5c11e511894fa6c11277668702a
Author: wangbo <wa...@apache.org>
AuthorDate: Mon Feb 26 18:07:11 2024 +0800
Fix cgroup cpu controller file leak (#31398)
---
be/src/agent/cgroup_cpu_ctl.cpp | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/be/src/agent/cgroup_cpu_ctl.cpp b/be/src/agent/cgroup_cpu_ctl.cpp
index ff44e5630e9..c94a3c05f1e 100644
--- a/be/src/agent/cgroup_cpu_ctl.cpp
+++ b/be/src/agent/cgroup_cpu_ctl.cpp
@@ -22,6 +22,8 @@
#include <filesystem>
+#include "util/defer_op.h"
+
namespace doris {
Status CgroupCpuCtl::init() {
@@ -84,6 +86,12 @@ Status CgroupCpuCtl::write_cg_sys_file(std::string file_path, int value, std::st
return Status::InternalError<false>("open path failed, path={}", file_path);
}
+ Defer defer {[&]() {
+ if (-1 == ::close(fd)) {
+ LOG(INFO) << "close file fd failed";
+ }
+ }};
+
auto str = fmt::format("{}\n", value);
int ret = write(fd, str.c_str(), str.size());
if (ret == -1) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org