You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2023/02/02 11:36:22 UTC
[doris] branch branch-1.2-lts updated (495d37d337 -> be7c4d267e)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
from 495d37d337 [fix](join) crash caused by canceling query (#16311) (#16349)
new e48404e1c0 [fix](planner) create view generate wrong sql when sql contains multi count distinct (#16092)
new 38c9fe7f8d [Refactor](function) opt the exec of function with null column (#16256)
new a1dcec461c [Enhance] use fast_float::from_chars to do str cast to float/double to avoid lose precision (#16190)
new 29e6480bc8 [fix](multi-catalog) remove the eof check among parquet columns (#16302)
new df7200f8ae [test](regression) add tvf regression to test the remove of eof check (#16342)
new be7c4d267e [branch1.2] fix compile bug after cherry-pick
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/util/string_parser.hpp | 146 ++++++---------------
.../exec/format/parquet/vparquet_group_reader.cpp | 9 +-
be/src/vec/exprs/vectorized_fn_call.cpp | 2 +
be/src/vec/functions/function.cpp | 14 +-
be/src/vec/functions/function_cast.h | 11 +-
be/src/vec/functions/function_helpers.cpp | 119 ++++++++---------
be/src/vec/functions/function_helpers.h | 26 ++--
be/test/util/string_parser_test.cpp | 5 +-
.../org/apache/doris/analysis/BaseViewStmt.java | 1 +
.../apache/doris/analysis/FunctionCallExpr.java | 1 +
.../java/org/apache/doris/analysis/SelectStmt.java | 8 +-
regression-test/conf/regression-conf.groovy | 29 ++++
.../external_table_emr_p2/hive/test_tvf_p2.out | 32 +++++
.../suites/ddl_p0/test_create_view.groovy | 72 ++++++++++
.../external_table_emr_p2/hive/test_tvf_p2.groovy | 31 ++---
15 files changed, 276 insertions(+), 230 deletions(-)
create mode 100644 regression-test/data/external_table_emr_p2/hive/test_tvf_p2.out
create mode 100644 regression-test/suites/ddl_p0/test_create_view.groovy
copy be/src/runtime/tuple_row.cpp => regression-test/suites/external_table_emr_p2/hive/test_tvf_p2.groovy (55%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 06/06: [branch1.2] fix compile bug after cherry-pick
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit be7c4d267e09d0b6f7ed6930684f270d1324f91d
Author: morningman <mo...@163.com>
AuthorDate: Thu Feb 2 18:11:55 2023 +0800
[branch1.2] fix compile bug after cherry-pick
---
be/src/vec/functions/function_helpers.cpp | 8 --------
.../src/main/java/org/apache/doris/analysis/FunctionCallExpr.java | 1 +
2 files changed, 1 insertion(+), 8 deletions(-)
diff --git a/be/src/vec/functions/function_helpers.cpp b/be/src/vec/functions/function_helpers.cpp
index c77f3c5ab7..c42d0dc0ea 100644
--- a/be/src/vec/functions/function_helpers.cpp
+++ b/be/src/vec/functions/function_helpers.cpp
@@ -79,14 +79,6 @@ std::tuple<Block, ColumnNumbers> create_block_with_nested_columns(const Block& b
}
}
- // TODO: only support match function, rethink the logic
- for (const auto& ctn : block) {
- if (ctn.name.size() > BeConsts::BLOCK_TEMP_COLUMN_PREFIX.size() &&
- starts_with(ctn.name, BeConsts::BLOCK_TEMP_COLUMN_PREFIX)) {
- res.insert(ctn);
- }
- }
-
return {res, res_args};
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index 526d2fd9f6..3aff4d3517 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -835,6 +835,7 @@ public class FunctionCallExpr extends Expr {
if (!getChild(1).isConstant()) {
throw new AnalysisException(fnName + "function's second argument should be constant");
}
+ throw new AnalysisException(fnName + "not support on vectorized engine now.");
}
if ((fnName.getFunction().equalsIgnoreCase("HLL_UNION_AGG")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/06: [fix](multi-catalog) remove the eof check among parquet columns (#16302)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 29e6480bc8be1fc882c3e7a1f28b7164a3b71c97
Author: Ashin Gau <As...@users.noreply.github.com>
AuthorDate: Thu Feb 2 09:22:09 2023 +0800
[fix](multi-catalog) remove the eof check among parquet columns (#16302)
Read parquet file failed:
```
ERROR 1105 (HY000): errCode = 2, detailMessage = [INTERNAL_ERROR]Read parquet file xxx failed, reason = [CORRUPTION]The number of rows are not equal among parquet columns
```
This error may be thrown when reading non-predicate columns in lazy-read, for example:
A row group with 1000 rows has tow non-predicate columns.
Column A has one page, Column B has two pages with 500 rows for each page.
The read range of `ParquetColumnReader` is [0, 400), and the rows between [0, 450) are all filtered by predicate columns.
So column A can skip the first page, and reach the EOF, while column B can also skip the first page, but doesn't read the EOF.
---
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp b/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
index 34b478114e..5b1c0fd828 100644
--- a/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_group_reader.cpp
@@ -134,7 +134,6 @@ Status RowGroupReader::_read_column_data(Block* block, const std::vector<std::st
ColumnSelectVector& select_vector) {
size_t batch_read_rows = 0;
bool has_eof = false;
- int col_idx = 0;
for (auto& read_col : columns) {
auto& column_with_type_and_name = block->get_by_name(read_col);
auto& column_ptr = column_with_type_and_name.column;
@@ -150,15 +149,13 @@ Status RowGroupReader::_read_column_data(Block* block, const std::vector<std::st
&col_eof));
col_read_rows += loop_rows;
}
- if (col_idx > 0 && (has_eof ^ col_eof)) {
- return Status::Corruption("The number of rows are not equal among parquet columns");
- }
if (batch_read_rows > 0 && batch_read_rows != col_read_rows) {
return Status::Corruption("Can't read the same number of rows among parquet columns");
}
batch_read_rows = col_read_rows;
- has_eof = col_eof;
- col_idx++;
+ if (col_eof) {
+ has_eof = true;
+ }
}
*read_rows = batch_read_rows;
*batch_eof = has_eof;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/06: [Refactor](function) opt the exec of function with null column (#16256)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 38c9fe7f8d7ec7e91c6821cd66c8d366e7d98bf0
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Feb 1 15:56:31 2023 +0800
[Refactor](function) opt the exec of function with null column (#16256)
---
be/src/vec/exprs/vectorized_fn_call.cpp | 2 +
be/src/vec/functions/function.cpp | 14 ++--
be/src/vec/functions/function_cast.h | 11 ++-
be/src/vec/functions/function_helpers.cpp | 123 ++++++++++++++----------------
be/src/vec/functions/function_helpers.h | 26 +++----
5 files changed, 86 insertions(+), 90 deletions(-)
diff --git a/be/src/vec/exprs/vectorized_fn_call.cpp b/be/src/vec/exprs/vectorized_fn_call.cpp
index 3999599716..a0e07d31f9 100644
--- a/be/src/vec/exprs/vectorized_fn_call.cpp
+++ b/be/src/vec/exprs/vectorized_fn_call.cpp
@@ -51,6 +51,8 @@ doris::Status VectorizedFnCall::prepare(doris::RuntimeState* state,
argument_template.reserve(_children.size());
std::vector<std::string_view> child_expr_name;
for (auto child : _children) {
+ // TODO: rethink we really create column here. maybe only need nullptr just to
+ // get the function
auto column = child->data_type()->create_column();
argument_template.emplace_back(std::move(column), child->data_type(), child->expr_name());
child_expr_name.emplace_back(child->expr_name());
diff --git a/be/src/vec/functions/function.cpp b/be/src/vec/functions/function.cpp
index 41f3141c06..662a2a58af 100644
--- a/be/src/vec/functions/function.cpp
+++ b/be/src/vec/functions/function.cpp
@@ -217,11 +217,12 @@ Status PreparedFunctionImpl::default_implementation_for_nulls(
}
if (null_presence.has_nullable) {
- Block temporary_block = create_block_with_nested_columns(block, args, result);
+ auto [temporary_block, new_args, new_result] =
+ create_block_with_nested_columns(block, args, result);
RETURN_IF_ERROR(execute_without_low_cardinality_columns(
- context, temporary_block, args, result, temporary_block.rows(), dry_run));
+ context, temporary_block, new_args, new_result, temporary_block.rows(), dry_run));
block.get_by_position(result).column =
- wrap_in_nullable(temporary_block.get_by_position(result).column, block, args,
+ wrap_in_nullable(temporary_block.get_by_position(new_result).column, block, args,
result, input_rows_count);
*executed = true;
return Status::OK();
@@ -295,10 +296,9 @@ DataTypePtr FunctionBuilderImpl::get_return_type_without_low_cardinality(
}
if (null_presence.has_nullable) {
ColumnNumbers numbers(arguments.size());
- for (size_t i = 0; i < arguments.size(); i++) {
- numbers[i] = i;
- }
- Block nested_block = create_block_with_nested_columns(Block(arguments), numbers);
+ std::iota(numbers.begin(), numbers.end(), 0);
+ auto [nested_block, _] =
+ create_block_with_nested_columns(Block(arguments), numbers, false);
auto return_type = get_return_type_impl(
ColumnsWithTypeAndName(nested_block.begin(), nested_block.end()));
return make_nullable(return_type);
diff --git a/be/src/vec/functions/function_cast.h b/be/src/vec/functions/function_cast.h
index e3baaecdd2..a6817134ea 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -1592,7 +1592,9 @@ private:
Block tmp_block;
size_t tmp_res_index = 0;
if (source_is_nullable) {
- tmp_block = create_block_with_nested_columns_only_args(block, arguments);
+ auto [t_block, tmp_args] =
+ create_block_with_nested_columns(block, arguments, true);
+ tmp_block = std::move(t_block);
tmp_res_index = tmp_block.columns();
tmp_block.insert({nullptr, nested_type, ""});
@@ -1624,7 +1626,8 @@ private:
return [wrapper, skip_not_null_check](FunctionContext* context, Block& block,
const ColumnNumbers& arguments,
const size_t result, size_t input_rows_count) {
- Block tmp_block = create_block_with_nested_columns(block, arguments, result);
+ auto [tmp_block, tmp_args, tmp_res] =
+ create_block_with_nested_columns(block, arguments, result);
/// Check that all values are not-NULL.
/// Check can be skipped in case if LowCardinality dictionary is transformed.
@@ -1640,8 +1643,8 @@ private:
}
}
- RETURN_IF_ERROR(wrapper(context, tmp_block, arguments, result, input_rows_count));
- block.get_by_position(result).column = tmp_block.get_by_position(result).column;
+ RETURN_IF_ERROR(wrapper(context, tmp_block, tmp_args, tmp_res, input_rows_count));
+ block.get_by_position(result).column = tmp_block.get_by_position(tmp_res).column;
return Status::OK();
};
} else {
diff --git a/be/src/vec/functions/function_helpers.cpp b/be/src/vec/functions/function_helpers.cpp
index fcfdd4b3a2..c77f3c5ab7 100644
--- a/be/src/vec/functions/function_helpers.cpp
+++ b/be/src/vec/functions/function_helpers.cpp
@@ -20,88 +20,83 @@
#include "vec/functions/function_helpers.h"
+#include "common/consts.h"
#include "vec/columns/column_nullable.h"
#include "vec/data_types/data_type_nullable.h"
#include "vec/functions/function.h"
namespace doris::vectorized {
-Block create_block_with_nested_columns_only_args(const Block& block, const ColumnNumbers& args) {
- std::set<size_t> args_set(args.begin(), args.end());
+std::tuple<Block, ColumnNumbers> create_block_with_nested_columns(const Block& block,
+ const ColumnNumbers& args,
+ const bool need_check_same) {
Block res;
+ ColumnNumbers res_args(args.size());
+
+ // only build temp block by args column, if args[i] == args[j]
+ // just keep one
+ for (size_t i = 0; i < args.size(); ++i) {
+ bool is_in_res = false;
+ size_t pre_loc = 0;
+
+ if (need_check_same) {
+ for (int j = 0; j < i; ++j) {
+ if (args[j] == args[i]) {
+ is_in_res = true;
+ pre_loc = res_args[j];
+ break;
+ }
+ }
+ }
- for (auto i : args_set) {
- const auto& col = block.get_by_position(i);
-
- if (col.type->is_nullable()) {
- const DataTypePtr& nested_type =
- static_cast<const DataTypeNullable&>(*col.type).get_nested_type();
-
- if (!col.column) {
- res.insert({nullptr, nested_type, col.name});
- } else if (auto* nullable = check_and_get_column<ColumnNullable>(*col.column)) {
- const auto& nested_col = nullable->get_nested_column_ptr();
- res.insert({nested_col, nested_type, col.name});
- } else if (auto* const_column = check_and_get_column<ColumnConst>(*col.column)) {
- const auto& nested_col =
- check_and_get_column<ColumnNullable>(const_column->get_data_column())
- ->get_nested_column_ptr();
- res.insert({ColumnConst::create(nested_col, col.column->size()), nested_type,
- col.name});
+ if (!is_in_res) {
+ const auto& col = block.get_by_position(args[i]);
+ if (col.type->is_nullable()) {
+ const DataTypePtr& nested_type =
+ static_cast<const DataTypeNullable&>(*col.type).get_nested_type();
+
+ if (!col.column) {
+ res.insert({nullptr, nested_type, col.name});
+ } else if (auto* nullable = check_and_get_column<ColumnNullable>(*col.column)) {
+ const auto& nested_col = nullable->get_nested_column_ptr();
+ res.insert({nested_col, nested_type, col.name});
+ } else if (auto* const_column = check_and_get_column<ColumnConst>(*col.column)) {
+ const auto& nested_col =
+ check_and_get_column<ColumnNullable>(const_column->get_data_column())
+ ->get_nested_column_ptr();
+ res.insert({ColumnConst::create(nested_col, col.column->size()), nested_type,
+ col.name});
+ } else {
+ LOG(FATAL) << "Illegal column for DataTypeNullable";
+ }
} else {
- LOG(FATAL) << "Illegal column for DataTypeNullable";
+ res.insert(col);
}
+
+ res_args[i] = res.columns() - 1;
} else {
- res.insert(col);
+ res_args[i] = pre_loc;
}
}
- return res;
-}
-
-static Block create_block_with_nested_columns_impl(const Block& block,
- const std::unordered_set<size_t>& args) {
- Block res;
- size_t columns = block.columns();
-
- for (size_t i = 0; i < columns; ++i) {
- const auto& col = block.get_by_position(i);
-
- if (args.count(i) && col.type->is_nullable()) {
- const DataTypePtr& nested_type =
- static_cast<const DataTypeNullable&>(*col.type).get_nested_type();
-
- if (!col.column) {
- res.insert({nullptr, nested_type, col.name});
- } else if (auto* nullable = check_and_get_column<ColumnNullable>(*col.column)) {
- const auto& nested_col = nullable->get_nested_column_ptr();
- res.insert({nested_col, nested_type, col.name});
- } else if (auto* const_column = check_and_get_column<ColumnConst>(*col.column)) {
- const auto& nested_col =
- check_and_get_column<ColumnNullable>(const_column->get_data_column())
- ->get_nested_column_ptr();
- res.insert({ColumnConst::create(nested_col, col.column->size()), nested_type,
- col.name});
- } else {
- LOG(FATAL) << "Illegal column for DataTypeNullable";
- }
- } else
- res.insert(col);
+ // TODO: only support match function, rethink the logic
+ for (const auto& ctn : block) {
+ if (ctn.name.size() > BeConsts::BLOCK_TEMP_COLUMN_PREFIX.size() &&
+ starts_with(ctn.name, BeConsts::BLOCK_TEMP_COLUMN_PREFIX)) {
+ res.insert(ctn);
+ }
}
- return res;
-}
-
-Block create_block_with_nested_columns(const Block& block, const ColumnNumbers& args) {
- std::unordered_set<size_t> args_set(args.begin(), args.end());
- return create_block_with_nested_columns_impl(block, args_set);
+ return {res, res_args};
}
-Block create_block_with_nested_columns(const Block& block, const ColumnNumbers& args,
- size_t result) {
- std::unordered_set<size_t> args_set(args.begin(), args.end());
- args_set.insert(result);
- return create_block_with_nested_columns_impl(block, args_set);
+std::tuple<Block, ColumnNumbers, size_t> create_block_with_nested_columns(const Block& block,
+ const ColumnNumbers& args,
+ size_t result) {
+ auto [res, res_args] = create_block_with_nested_columns(block, args, true);
+ // insert result column in temp block
+ res.insert(block.get_by_position(result));
+ return {res, res_args, res.columns() - 1};
}
void validate_argument_type(const IFunction& func, const DataTypes& arguments,
diff --git a/be/src/vec/functions/function_helpers.h b/be/src/vec/functions/function_helpers.h
index 777bd415dc..474ac435bf 100644
--- a/be/src/vec/functions/function_helpers.h
+++ b/be/src/vec/functions/function_helpers.h
@@ -86,21 +86,17 @@ inline std::enable_if_t<IsDecimalNumber<T>, Field> to_field(const T& x, UInt32 s
Columns convert_const_tuple_to_constant_elements(const ColumnConst& column);
-/// Returns the copy of a given block in which each column specified in
-/// the "arguments" parameter is replaced with its respective nested
-/// column if it is nullable.
-Block create_block_with_nested_columns(const Block& block, const ColumnNumbers& args);
-
-/// Similar function as above. Additionally transform the result type if needed.
-Block create_block_with_nested_columns(const Block& block, const ColumnNumbers& args,
- size_t result);
-
-/// Returns the copy of a given block in only args column specified in
-/// the "arguments" parameter is replaced with its respective nested
-/// column if it is nullable.
-/// TODO: the old funciton `create_block_with_nested_columns` have performance problem, replace all
-/// by the function and delete old one.
-Block create_block_with_nested_columns_only_args(const Block& block, const ColumnNumbers& args);
+/// Returns the copy of a tmp block and temp args order same as args
+/// in which only args column each column specified in the "arguments"
+/// parameter is replaced with its respective nested column if it is nullable.
+std::tuple<Block, ColumnNumbers> create_block_with_nested_columns(const Block& block,
+ const ColumnNumbers& args,
+ const bool need_check_same);
+
+// Same as above and return the new_res loc in tuple
+std::tuple<Block, ColumnNumbers, size_t> create_block_with_nested_columns(const Block& block,
+ const ColumnNumbers& args,
+ size_t result);
/// Checks argument type at specified index with predicate.
/// throws if there is no argument at specified index or if predicate returns false.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 05/06: [test](regression) add tvf regression to test the remove of eof check (#16342)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit df7200f8aeacead5b42ec9b74b528dab2a806675
Author: Ashin Gau <As...@users.noreply.github.com>
AuthorDate: Thu Feb 2 10:06:36 2023 +0800
[test](regression) add tvf regression to test the remove of eof check (#16342)
Add regression test for #16302. This regression test will be failed if add EOF check for non-predicate columns.
---
regression-test/conf/regression-conf.groovy | 29 ++++++++++++++++++++
.../external_table_emr_p2/hive/test_tvf_p2.out | 32 ++++++++++++++++++++++
.../external_table_emr_p2/hive/test_tvf_p2.groovy | 30 ++++++++++++++++++++
3 files changed, 91 insertions(+)
diff --git a/regression-test/conf/regression-conf.groovy b/regression-test/conf/regression-conf.groovy
index 544d4b12c1..6779873340 100644
--- a/regression-test/conf/regression-conf.groovy
+++ b/regression-test/conf/regression-conf.groovy
@@ -94,6 +94,35 @@ es_8_port=39200
cacheDataPath = "/tmp"
+//hive catalog test config for bigdata
+enableExternalHiveTest = false
+extHiveHmsHost = "***.**.**.**"
+extHiveHmsPort = 7004
+extHdfsPort = 4007
+extHiveHmsUser = "****"
+extHiveHmsPassword= "***********"
+
+//mysql jdbc connector test config for bigdata
+enableExternalMysqlTest = false
+extMysqlHost = "***.**.**.**"
+extMysqlPort = 3306
+extMysqlUser = "****"
+extMysqlPassword = "***********"
+
+//postgresql jdbc connector test config for bigdata
+enableExternalPgTest = false
+extPgHost = "***.**.**.*"
+extPgPort = 5432
+extPgUser = "****"
+extPgPassword = "***********"
+
+// elasticsearch external test config for bigdata
+enableExternalEsTest = false
+extEsHost = "***********"
+extEsPort = 9200
+extEsUser = "*******"
+extEsPassword = "***********"
+
s3Endpoint = "cos.ap-hongkong.myqcloud.com"
s3BucketName = "doris-build-hk-1308700295"
s3Region = "ap-hongkong"
diff --git a/regression-test/data/external_table_emr_p2/hive/test_tvf_p2.out b/regression-test/data/external_table_emr_p2/hive/test_tvf_p2.out
new file mode 100644
index 0000000000..7f94b13974
--- /dev/null
+++ b/regression-test/data/external_table_emr_p2/hive/test_tvf_p2.out
@@ -0,0 +1,32 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !eof_check --
+2451718 \N 9242 \N \N 2886 \N 4 250 1374252 18 \N \N \N 0 1513 1435 \N \N 0 \N 1588 78
+\N \N 14846 1945858 \N 1015 \N 4 581 2383831 \N \N 5 1 0 110 \N \N \N 0 110 \N -213
+\N 50835 25618 1166535 \N 1748 \N 4 \N 2880907 7 \N 17 \N \N 115 \N 125 1 \N 115 \N \N
+2452195 45280 29385 1298621 1649018 1815 \N 4 \N 3379765 24 73 \N \N 0 2617 1770 3399 \N 0 \N 2826 \N
+2451488 53117 31945 \N 8644 \N \N 4 783 4877135 100 \N \N \N \N \N 3450 \N \N \N 565 581 -2885
+\N 53900 35887 702626 \N 2568 \N 4 \N 2381514 \N \N \N 0 \N \N \N \N 1 \N 19 20 -357
+\N 53985 38881 760602 289764 \N \N 4 227 3377513 68 75 \N \N \N 5833 \N \N \N \N 524 \N -4588
+\N \N 51685 1833943 \N \N \N 4 \N 1879197 \N \N \N \N 0 46 163 \N \N 0 \N 49 -116
+\N \N 62073 \N 287578 \N \N 4 990 16264789 90 91 \N \N 0 6381 8247 \N \N 0 6381 \N \N
+\N 34914 64259 167395 897626 \N \N 4 327 19379058 15 \N \N 51 \N \N 1480 \N \N \N \N \N -707
+\N 70509 100949 \N \N \N \N 4 185 2381361 35 1 \N \N 0 \N 41 \N \N 0 \N 82 33
+2452489 74165 103575 \N 1359778 \N \N 4 \N 2383538 1 \N 23 0 \N 0 15 23 \N \N 0 \N -14
+2451253 \N 111502 246668 \N \N \N 4 \N 2881367 \N \N \N 21 0 \N \N 1218 74 0 \N 999 -49
+2451093 \N 121339 \N \N \N \N 4 894 19379088 11 92 \N \N \N \N \N 1364 9 \N 305 314 \N
+2452592 \N 126241 1673449 1682209 \N \N 4 \N 16872613 \N 4 7 4 0 20 20 \N \N 0 \N 21 0
+2451985 \N 128921 \N \N \N \N 4 \N 17767165 \N \N \N \N \N \N \N \N 55 \N 929 \N -1765
+\N \N 137292 1348279 1465562 6398 \N 4 922 372083 \N \N 130 65 \N 3772 3772 7544 \N \N \N 4036 \N
+\N \N 146095 494197 \N 2565 \N 4 352 17767165 \N \N 24 22 776 \N 1877 2158 \N 776 1165 \N \N
+\N \N 158078 889184 \N \N \N 4 \N 369514 \N 31 48 31 0 501 \N \N 20 0 \N \N \N
+\N \N 159005 397184 1836520 3856 \N 4 \N 371981 \N \N \N 0 0 0 \N 208 0 0 0 0 -163
+\N 70509 160172 \N 1578153 5827 \N 4 \N 2381361 32 \N \N 1 \N 56 3037 \N \N \N 56 61 \N
+\N \N 166592 1688819 891341 \N \N 4 535 1879015 38 77 125 56 \N 2148 \N \N 107 \N \N 2256 \N
+2452394 45881 167121 1409858 294280 \N \N 4 752 2381343 63 \N \N \N \N \N 2271 4473 \N \N 2460 \N \N
+\N 58568 174392 742421 \N \N \N 4 104 19379109 100 75 93 \N \N \N \N \N 54 \N \N 5494 -2125
+2451460 71227 178990 \N \N \N \N 4 \N 2878825 22 59 \N 10 0 \N 1317 \N 9 0 239 \N \N
+\N \N 196777 \N \N \N \N 4 954 2381557 84 \N 4 \N \N \N 216 \N \N \N 142 \N -74
+\N \N 198073 \N 1537282 3742 \N 4 \N 3878103 31 \N \N 7 \N 239 \N 615 \N \N \N \N -370
+2451656 \N 200558 \N \N 1066 \N 4 \N 2381557 \N 79 \N 94 0 \N \N 12956 481 0 \N 8514 1248
+\N \N 203791 \N 1655274 6679 \N 4 \N 3379960 71 \N 96 45 \N 3214 3525 6840 160 \N \N \N \N
+
diff --git a/regression-test/suites/external_table_emr_p2/hive/test_tvf_p2.groovy b/regression-test/suites/external_table_emr_p2/hive/test_tvf_p2.groovy
new file mode 100644
index 0000000000..660aa7a60b
--- /dev/null
+++ b/regression-test/suites/external_table_emr_p2/hive/test_tvf_p2.groovy
@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_tvf_p2", "p2") {
+ String enabled = context.config.otherConfigs.get("enableExternalHiveTest")
+ if (enabled != null && enabled.equalsIgnoreCase("true")) {
+ String nameNodeHost = context.config.otherConfigs.get("extHiveHmsHost")
+ String hdfsPort = context.config.otherConfigs.get("extHdfsPort")
+
+ qt_eof_check """select * from hdfs(
+ "uri" = "hdfs://${nameNodeHost}:${hdfsPort}/user/data/tvf/bad_store_sales.parquet",
+ "fs.defaultFS" = "hdfs://${nameNodeHost}:${hdfsPort}",
+ "format" = "parquet")
+ where ss_store_sk = 4 and ss_addr_sk is null order by ss_item_sk"""
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/06: [Enhance] use fast_float::from_chars to do str cast to float/double to avoid lose precision (#16190)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a1dcec461cd17f734ec7cff349328a1dcf96f55d
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Feb 1 23:53:34 2023 +0800
[Enhance] use fast_float::from_chars to do str cast to float/double to avoid lose precision (#16190)
---
be/src/util/string_parser.hpp | 146 ++++++++++--------------------------
be/test/util/string_parser_test.cpp | 5 +-
2 files changed, 41 insertions(+), 110 deletions(-)
diff --git a/be/src/util/string_parser.hpp b/be/src/util/string_parser.hpp
index 653f0dac14..02006b7c7d 100644
--- a/be/src/util/string_parser.hpp
+++ b/be/src/util/string_parser.hpp
@@ -20,6 +20,8 @@
#pragma once
+#include <fast_float/fast_float.h>
+
#include <cmath>
#include <cstdint>
#include <cstring>
@@ -111,13 +113,7 @@ public:
template <typename T>
static inline T string_to_float(const char* s, int len, ParseResult* result) {
- T ans = string_to_float_internal<T>(s, len, result);
- if (LIKELY(*result == PARSE_SUCCESS)) {
- return ans;
- }
-
- int i = skip_leading_whitespace(s, len);
- return string_to_float_internal<T>(s + i, len - i, result);
+ return string_to_float_internal<T>(s, len, result);
}
// Parses a string for 'true' or 'false', case insensitive.
@@ -425,118 +421,54 @@ inline T StringParser::string_to_int_no_overflow(const char* s, int len, ParseRe
template <typename T>
inline T StringParser::string_to_float_internal(const char* s, int len, ParseResult* result) {
- if (UNLIKELY(len <= 0)) {
+ int i = 0;
+ // skip leading spaces
+ for (; i < len; ++i) {
+ if (!is_whitespace(s[i])) {
+ break;
+ }
+ }
+
+ // skip back spaces
+ int j = len - 1;
+ for (; j >= i; j--) {
+ if (!is_whitespace(s[j])) {
+ break;
+ }
+ }
+
+ // skip leading '+', from_chars can handle '-'
+ if (i < len && s[i] == '+') {
+ i++;
+ }
+ if (UNLIKELY(i > j)) {
*result = PARSE_FAILURE;
return 0;
}
// Use double here to not lose precision while accumulating the result
double val = 0;
- bool negative = false;
- int i = 0;
- double divide = 1;
- bool decimal = false;
- int64_t remainder = 0;
- // The number of 'significant figures' we've encountered so far (i.e., digits excluding
- // leading 0s). This technically shouldn't count trailing 0s either, but for us it
- // doesn't matter if we count them based on the implementation below.
- int sig_figs = 0;
-
- switch (*s) {
- case '-':
- negative = true;
- case '+':
- i = 1;
- }
-
- int first = i;
- for (; i < len; ++i) {
- if (LIKELY(s[i] >= '0' && s[i] <= '9')) {
- if (s[i] != '0' || sig_figs > 0) {
- ++sig_figs;
- }
- if (decimal) {
- // According to the IEEE floating-point spec, a double has up to 15-17
- // significant decimal digits (see
- // http://en.wikipedia.org/wiki/Double-precision_floating-point_format). We stop
- // processing digits after we've already seen at least 18 sig figs to avoid
- // overflowing 'remainder' (we stop after 18 instead of 17 to get the rounding
- // right).
- if (sig_figs <= 18) {
- remainder = remainder * 10 + s[i] - '0';
- divide *= 10;
+ auto res = fast_float::from_chars(s + i, s + j + 1, val);
+
+ if (res.ec == std::errc() && res.ptr == s + j + 1) {
+ if (abs(val) == std::numeric_limits<T>::infinity()) {
+ auto contain_inf = false;
+ for (int k = i; k < j + 1; k++) {
+ if (s[k] == 'i' || s[k] == 'I') {
+ contain_inf = true;
+ break;
}
- } else {
- val = val * 10 + s[i] - '0';
- }
- } else if (s[i] == '.') {
- decimal = true;
- } else if (s[i] == 'e' || s[i] == 'E') {
- break;
- } else if (s[i] == 'i' || s[i] == 'I') {
- if (len > i + 2 && (s[i + 1] == 'n' || s[i + 1] == 'N') &&
- (s[i + 2] == 'f' || s[i + 2] == 'F')) {
- // Note: Hive writes inf as Infinity, at least for text. We'll be a little loose
- // here and interpret any column with inf as a prefix as infinity rather than
- // checking every remaining byte.
- *result = PARSE_SUCCESS;
- return negative ? -INFINITY : INFINITY;
- } else {
- // Starts with 'i', but isn't inf...
- *result = PARSE_FAILURE;
- return 0;
- }
- } else if (s[i] == 'n' || s[i] == 'N') {
- if (len > i + 2 && (s[i + 1] == 'a' || s[i + 1] == 'A') &&
- (s[i + 2] == 'n' || s[i + 2] == 'N')) {
- *result = PARSE_SUCCESS;
- return negative ? -NAN : NAN;
- } else {
- // Starts with 'n', but isn't NaN...
- *result = PARSE_FAILURE;
- return 0;
- }
- } else {
- if ((UNLIKELY(i == first || !is_all_whitespace(s + i, len - i)))) {
- // Reject the string because either the first char was not a digit, "," or "e",
- // or the remaining chars are not all whitespace
- *result = PARSE_FAILURE;
- return 0;
}
- // skip trailing whitespace.
- break;
- }
- }
- val += remainder / divide;
-
- if (i < len && (s[i] == 'e' || s[i] == 'E')) {
- // Create a C-string from s starting after the optional '-' sign and fall back to
- // strtod to avoid conversion inaccuracy for scientific notation.
- // Do not use boost::lexical_cast because it causes codegen to crash for an
- // unknown reason (exception handling?).
- char c_str[len - negative + 1];
- memcpy(c_str, s + negative, len - negative);
- c_str[len - negative] = '\0';
- char* s_end;
- val = strtod(c_str, &s_end);
- if (s_end != c_str + len - negative) {
- // skip trailing whitespace
- int trailing_len = len - negative - (int)(s_end - c_str);
- if (UNLIKELY(!is_all_whitespace(s_end, trailing_len))) {
- *result = PARSE_FAILURE;
- return val;
- }
+ *result = contain_inf ? PARSE_SUCCESS : PARSE_OVERFLOW;
+ } else {
+ *result = PARSE_SUCCESS;
}
- }
-
- // Determine if it is an overflow case and update the result
- if (UNLIKELY(val == std::numeric_limits<T>::infinity())) {
- *result = PARSE_OVERFLOW;
+ return val;
} else {
- *result = PARSE_SUCCESS;
+ *result = PARSE_FAILURE;
}
- return (T)(negative ? -val : val);
+ return 0;
}
inline bool StringParser::string_to_bool_internal(const char* s, int len, ParseResult* result) {
diff --git a/be/test/util/string_parser_test.cpp b/be/test/util/string_parser_test.cpp
index a85e5b97ed..f9e9c37050 100644
--- a/be/test/util/string_parser_test.cpp
+++ b/be/test/util/string_parser_test.cpp
@@ -441,7 +441,6 @@ TEST(StringToFloat, Basic) {
std::string double_max = boost::lexical_cast<std::string>(std::numeric_limits<double>::max());
test_float_value<double>(double_min, StringParser::PARSE_SUCCESS);
test_float_value<double>(double_max, StringParser::PARSE_SUCCESS);
-
// Non-finite values
test_all_float_variants("INFinity", StringParser::PARSE_SUCCESS);
test_all_float_variants("infinity", StringParser::PARSE_SUCCESS);
@@ -451,8 +450,8 @@ TEST(StringToFloat, Basic) {
test_float_value_is_nan<double>("nan", StringParser::PARSE_SUCCESS);
test_float_value_is_nan<float>("NaN", StringParser::PARSE_SUCCESS);
test_float_value_is_nan<double>("NaN", StringParser::PARSE_SUCCESS);
- test_float_value_is_nan<float>("nana", StringParser::PARSE_SUCCESS);
- test_float_value_is_nan<double>("nana", StringParser::PARSE_SUCCESS);
+ test_float_value_is_nan<float>("nana", StringParser::PARSE_FAILURE);
+ test_float_value_is_nan<double>("nana", StringParser::PARSE_FAILURE);
test_float_value_is_nan<float>("naN", StringParser::PARSE_SUCCESS);
test_float_value_is_nan<double>("naN", StringParser::PARSE_SUCCESS);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/06: [fix](planner) create view generate wrong sql when sql contains multi count distinct (#16092)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e48404e1c08f4887506181a4b6dd577cc4dc534e
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Tue Jan 31 23:42:53 2023 +0800
[fix](planner) create view generate wrong sql when sql contains multi count distinct (#16092)
If sql in create view has more than one count distinct, and write column name explicitly.
We will generate sql contains function multi_count_distinct.
It cannot be analyzed and all query containing this view will fail.
---
.../org/apache/doris/analysis/BaseViewStmt.java | 1 +
.../java/org/apache/doris/analysis/SelectStmt.java | 8 ++-
.../suites/ddl_p0/test_create_view.groovy | 72 ++++++++++++++++++++++
3 files changed, 80 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/BaseViewStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/BaseViewStmt.java
index 477e440f5e..8114448f0d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BaseViewStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BaseViewStmt.java
@@ -117,6 +117,7 @@ public class BaseViewStmt extends DdlStmt {
Analyzer tmpAnalyzer = new Analyzer(analyzer);
List<String> colNames = cols.stream().map(c -> c.getColName()).collect(Collectors.toList());
+ cloneStmt.setNeedToSql(true);
cloneStmt.substituteSelectList(tmpAnalyzer, colNames);
try (ToSqlContext toSqlContext = ToSqlContext.getOrNewThreadLocalContext()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 1aae7c324f..8f47a9ed04 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -1898,7 +1898,7 @@ public class SelectStmt extends QueryStmt {
if (i != 0) {
strBuilder.append(", ");
}
- if (needToSql) {
+ if (needToSql && CollectionUtils.isNotEmpty(originalExpr)) {
strBuilder.append(originalExpr.get(i).toSql());
} else {
strBuilder.append(resultExprs.get(i).toSql());
@@ -2072,6 +2072,9 @@ public class SelectStmt extends QueryStmt {
// Resolve and replace non-InlineViewRef table refs with a BaseTableRef or ViewRef.
TableRef tblRef = fromClause.get(i);
tblRef = analyzer.resolveTableRef(tblRef);
+ if (tblRef instanceof InlineViewRef) {
+ ((InlineViewRef) tblRef).setNeedToSql(needToSql);
+ }
Preconditions.checkNotNull(tblRef);
fromClause.set(i, tblRef);
tblRef.setLeftTblRef(leftTblRef);
@@ -2101,6 +2104,9 @@ public class SelectStmt extends QueryStmt {
resultExprs.add(item.getExpr());
}
}
+ if (needToSql) {
+ originalExpr = Expr.cloneList(resultExprs);
+ }
// substitute group by
if (groupByClause != null) {
boolean aliasFirst = false;
diff --git a/regression-test/suites/ddl_p0/test_create_view.groovy b/regression-test/suites/ddl_p0/test_create_view.groovy
new file mode 100644
index 0000000000..4c401017ee
--- /dev/null
+++ b/regression-test/suites/ddl_p0/test_create_view.groovy
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_create_view") {
+
+ sql """DROP TABLE IF EXISTS count_distinct"""
+ sql """
+ CREATE TABLE IF NOT EXISTS count_distinct
+ (
+ RQ DATE NOT NULL COMMENT "日期",
+ v1 VARCHAR(100) NOT NULL COMMENT "字段1",
+ v2 VARCHAR(100) NOT NULL COMMENT "字段2",
+ v3 VARCHAR(100) REPLACE_IF_NOT_NULL COMMENT "字段3"
+ )
+ AGGREGATE KEY(RQ,v1,v2)
+ PARTITION BY RANGE(RQ)
+ (
+ PARTITION p20220908 VALUES LESS THAN ('2022-09-09')
+ )
+ DISTRIBUTED BY HASH(v1,v2) BUCKETS 3
+ PROPERTIES(
+ "replication_num" = "1",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.start" = "-3",
+ "dynamic_partition.end" = "3",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "3"
+ );
+ """
+ sql """
+ CREATE VIEW IF NOT EXISTS test_count_distinct
+ (
+ RQ comment "日期",
+ v1 comment "v1",
+ v2 comment "v2",
+ v3 comment "v3"
+ )
+ AS
+ select aa.RQ as RQ, aa.v1 as v1,aa.v2 as v2 , bb.v3 as v3 from
+ (
+ select RQ, count(distinct v1) as v1 , count(distinct v2 ) as v2
+ from count_distinct
+ group by RQ
+ ) aa
+ LEFT JOIN
+ (
+ select RQ, max(v3) as v3
+ from count_distinct
+ group by RQ
+ ) bb
+ on aa.RQ = bb.RQ;
+ """
+
+ sql """select * from test_count_distinct"""
+ sql """DROP VIEW IF EXISTS test_count_distinct"""
+ sql """DROP TABLE IF EXISTS count_distinct"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org