You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/12/29 13:42:59 UTC
[GitHub] [doris] eldenmoon opened a new pull request, #15491: Pq opt
eldenmoon opened a new pull request, #15491:
URL: https://github.com/apache/doris/pull/15491
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15491: Pq opt
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1060263567
##########
be/src/util/jsonb_document.h:
##########
@@ -168,29 +171,17 @@ class JsonbDocument {
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator==(const JsonbDocument& other) const { assert(false); }
Review Comment:
warning: non-void function does not return a value [clang-diagnostic-return-type]
```cpp
bool operator==(const JsonbDocument& other) const { assert(false); }
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,29 +171,17 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator==(const JsonbDocument& other) const { assert(false); }
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator!=(const JsonbDocument& other) const { assert(false); }
Review Comment:
warning: non-void function does not return a value [clang-diagnostic-return-type]
```cpp
bool operator!=(const JsonbDocument& other) const { assert(false); }
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,29 +171,17 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator==(const JsonbDocument& other) const { assert(false); }
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator!=(const JsonbDocument& other) const { assert(false); }
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator<=(const JsonbDocument& other) const { assert(false); }
Review Comment:
warning: non-void function does not return a value [clang-diagnostic-return-type]
```cpp
bool operator<=(const JsonbDocument& other) const { assert(false); }
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,29 +171,17 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator==(const JsonbDocument& other) const { assert(false); }
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator!=(const JsonbDocument& other) const { assert(false); }
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator<=(const JsonbDocument& other) const { assert(false); }
- bool operator>=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator>=(const JsonbDocument& other) const { assert(false); }
Review Comment:
warning: non-void function does not return a value [clang-diagnostic-return-type]
```cpp
bool operator>=(const JsonbDocument& other) const { assert(false); }
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
Review Comment:
warning: calling a private constructor of class 'doris::SlotDescriptor' [clang-diagnostic-error]
```cpp
SlotDescriptor* slot = new SlotDescriptor(tslot1);
^
```
**be/src/runtime/descriptors.h:154:** declared private here
```cpp
SlotDescriptor(const TSlotDescriptor& tdesc);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
Review Comment:
warning: std::move of the const expression has no effect; remove std::move() [performance-move-const-arg]
```suggestion
auto mutable_nullable_vector = *column_nullable_vector.mutate();
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
Review Comment:
warning: calling a private constructor of class 'doris::SlotDescriptor' [clang-diagnostic-error]
```cpp
SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
^
```
**be/src/runtime/descriptors.h:154:** declared private here
```cpp
SlotDescriptor(const TSlotDescriptor& tdesc);
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,29 +171,17 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator==(const JsonbDocument& other) const { assert(false); }
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator!=(const JsonbDocument& other) const { assert(false); }
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator<=(const JsonbDocument& other) const { assert(false); }
- bool operator>=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator>=(const JsonbDocument& other) const { assert(false); }
- bool operator<(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator<(const JsonbDocument& other) const { assert(false); }
- bool operator>(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator>(const JsonbDocument& other) const { assert(false); }
Review Comment:
warning: non-void function does not return a value [clang-diagnostic-return-type]
```cpp
bool operator>(const JsonbDocument& other) const { assert(false); }
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,29 +171,17 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator==(const JsonbDocument& other) const { assert(false); }
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator!=(const JsonbDocument& other) const { assert(false); }
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator<=(const JsonbDocument& other) const { assert(false); }
- bool operator>=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator>=(const JsonbDocument& other) const { assert(false); }
- bool operator<(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
- }
+ bool operator<(const JsonbDocument& other) const { assert(false); }
Review Comment:
warning: non-void function does not return a value [clang-diagnostic-return-type]
```cpp
bool operator<(const JsonbDocument& other) const { assert(false); }
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ for (auto t : cols) {
+ TSlotDescriptor tslot;
+ tslot.__set_colName(std::get<0>(t));
+ if (std::get<3>(t) == TYPE_DECIMAL128I) {
+ TypeDescriptor type_desc(std::get<3>(t));
+ type_desc.precision = 27;
+ type_desc.scale = 9;
+ tslot.__set_slotType(type_desc.to_thrift());
+ } else {
+ TypeDescriptor type_desc(std::get<3>(t));
+ tslot.__set_slotType(type_desc.to_thrift());
+ }
+ tslot.__set_col_unique_id(std::get<2>(t));
+ SlotDescriptor* slot = new SlotDescriptor(tslot);
Review Comment:
warning: calling a private constructor of class 'doris::SlotDescriptor' [clang-diagnostic-error]
```cpp
SlotDescriptor* slot = new SlotDescriptor(tslot);
^
```
**be/src/runtime/descriptors.h:154:** declared private here
```cpp
SlotDescriptor(const TSlotDescriptor& tdesc);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ for (auto t : cols) {
+ TSlotDescriptor tslot;
+ tslot.__set_colName(std::get<0>(t));
+ if (std::get<3>(t) == TYPE_DECIMAL128I) {
+ TypeDescriptor type_desc(std::get<3>(t));
+ type_desc.precision = 27;
+ type_desc.scale = 9;
+ tslot.__set_slotType(type_desc.to_thrift());
+ } else {
+ TypeDescriptor type_desc(std::get<3>(t));
+ tslot.__set_slotType(type_desc.to_thrift());
+ }
+ tslot.__set_col_unique_id(std::get<2>(t));
+ SlotDescriptor* slot = new SlotDescriptor(tslot);
+ read_desc.add_slot(slot);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot);
^
```
**be/src/runtime/descriptors.h:380:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: calling a private constructor of class 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:378:** declared private here
```cpp
TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #15491:
URL: https://github.com/apache/doris/pull/15491#issuecomment-1369741004
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.81 seconds
load time: 651 seconds
storage size: 17123385668 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230103130055_clickbench_pr_72917.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
xiaokang commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1070872042
##########
fe/fe-core/src/main/cup/sql_parser.cup:
##########
@@ -6113,6 +6160,8 @@ literal ::=
{: RESULT = new BoolLiteral(false); :}
| KW_NULL
{: RESULT = new NullLiteral(); :}
+ | PLACEHOLDER
Review Comment:
what's the usage of PLACEHOLDER
##########
gensrc/proto/internal_service.proto:
##########
@@ -232,6 +232,38 @@ message PFetchDataResult {
optional bool empty_batch = 6;
};
+message KeyTuple {
+ repeated string key_column_rep = 1;
+}
+
+message UUID {
+ required int64 uuid_high = 1;
+ required int64 uuid_low = 2;
+}
+
+// We use thrift definition for some structure, since TExpr,
+// list<Exprs.TExpr>, Descriptors.TDescriptorTable are all thrift format.
+// Modify them to protobuf is a redundant work.
+message PTabletKeyLookupRequest {
+ required int64 tablet_id = 1;
+ repeated KeyTuple key_tuples = 2;
+
+ // reusable structures
+ // serilized from Descriptors.TDescriptorTable
+ optional UUID uuid = 3;
+ optional bytes desc_tbl = 4;
+ // serilized from TExprList
+ optional bytes output_expr = 5;
+ // return binary mysql row format if true
+ optional bool is_binary_row = 6;
+}
+
+message PTabletKeyLookupResponse {
+ required PStatus status = 1;
+ optional bytes row_batch = 5;
Review Comment:
sequence 5 is strange
##########
gensrc/proto/internal_service.proto:
##########
@@ -232,6 +232,38 @@ message PFetchDataResult {
optional bool empty_batch = 6;
};
+message KeyTuple {
+ repeated string key_column_rep = 1;
+}
+
+message UUID {
+ required int64 uuid_high = 1;
+ required int64 uuid_low = 2;
+}
+
+// We use thrift definition for some structure, since TExpr,
+// list<Exprs.TExpr>, Descriptors.TDescriptorTable are all thrift format.
+// Modify them to protobuf is a redundant work.
+message PTabletKeyLookupRequest {
+ required int64 tablet_id = 1;
+ repeated KeyTuple key_tuples = 2;
+
+ // reusable structures
+ // serilized from Descriptors.TDescriptorTable
+ optional UUID uuid = 3;
+ optional bytes desc_tbl = 4;
Review Comment:
why use bytes instead of real data structure for desc_tbl and output_expr?
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1648,10 +1721,37 @@ private void sendMetaData(ResultSetMetaData metaData) throws IOException {
context.getMysqlChannel().sendOnePacket(serializer.toByteBuffer());
}
+ private void sendStmtPrepareOK() throws IOException {
+ // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_prepare.html#sect_protocol_com_stmt_prepare_response
+ serializer.reset();
+ // 0x00 OK
+ serializer.writeInt1(0);
+ // statement_id
+ serializer.writeInt4(Integer.valueOf(prepareStmt.getName()));
Review Comment:
Is name guaranteed to be an int string?
##########
fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java:
##########
@@ -338,7 +339,6 @@ private void pushSortToOlapScan() {
PlanNode node = fragment.getPlanRoot();
PlanNode parent = null;
- // OlapScanNode is the last node.
Review Comment:
why delete this comment?
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3308,6 +3317,11 @@ public long getNextId() {
return idGenerator.getNextId();
}
+ // counter for prepared statement id
+ public long getNextStmtId() {
+ return this.stmtIdCounter.getAndIncrement();
Review Comment:
Is stmtIdCounter reset to zero on fe restart?
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/LiteralExpr.java:
##########
@@ -262,4 +262,106 @@ public boolean isNullable() {
public void finalizeImplForNereids() throws AnalysisException {
}
+
+ // Parse from binary data, the format follows mysql binary protocal
+ // see https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_binary_resultset.html.
+ // Return next offset
+ public void setupParamFromBinary(ByteBuffer data) {
+ Preconditions.checkState(false,
+ "should implement this in derived class. " + this.type.toSql());
+ }
+
+ public static LiteralExpr getLiteralByMysqlType(int mysqlType) throws AnalysisException {
+ switch (mysqlType) {
+ // MYSQL_TYPE_TINY
+ case 1:
+ return LiteralExpr.create("0", Type.TINYINT);
+ // MYSQL_TYPE_SHORT
+ case 2:
+ return LiteralExpr.create("0", Type.SMALLINT);
+ // MYSQL_TYPE_LONG
+ case 3:
+ return LiteralExpr.create("0", Type.INT);
+ // MYSQL_TYPE_LONGLONG
+ case 8:
+ return LiteralExpr.create("0", Type.BIGINT);
+ // MYSQL_TYPE_FLOAT
+ case 4:
+ return LiteralExpr.create("0", Type.FLOAT);
+ // MYSQL_TYPE_DOUBLE
+ case 5:
+ return LiteralExpr.create("0", Type.DOUBLE);
+ // MYSQL_TYPE_DECIMAL
+ case 0:
+ // MYSQL_TYPE_NEWDECIMAL
+ case 246:
+ return LiteralExpr.create("0", Type.DECIMAL32);
+ // MYSQL_TYPE_TIME
+ case 11:
+ return LiteralExpr.create("", Type.TIME);
+ // MYSQL_TYPE_DATE
+ case 10:
+ return LiteralExpr.create("1970-01-01", Type.DATE);
Review Comment:
DATE or DATEV2 ?
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java:
##########
@@ -1658,4 +1658,49 @@ public void setMinValue() {
second = 0;
microsecond = 0;
}
+
+ @Override
+ public void setupParamFromBinary(ByteBuffer data) {
+ int len = getParmLen(data);
+ if (type.getPrimitiveType() == PrimitiveType.DATE) {
+ if (len >= 4) {
+ year = (int) data.getChar();
+ month = (int) data.get();
+ day = (int) data.get();
+ hour = 0;
+ minute = 0;
+ second = 0;
+ microsecond = 0;
+ } else {
+ copy(MIN_DATE);
+ }
+ return;
+ }
+ if (type.getPrimitiveType() == PrimitiveType.DATETIME) {
Review Comment:
Is DATEV2 and DATETIMEV2 need to be processed?
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -155,6 +157,7 @@ public void setUserQueryTimeout(long queryTimeout) {
}
private StatementContext statementContext;
+ private Map<String, PrepareStmtContext> preparedStmtCtxs = Maps.newHashMap();
Review Comment:
If key shoud be a int string, using Integer as key is better.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -853,10 +908,13 @@ private void analyzeAndGenerateQueryPlan(TQueryOptions tQueryOptions) throws Use
}
List<String> origColLabels =
Lists.newArrayList(parsedStmt.getColLabels());
-
// Re-analyze the stmt with a new analyzer.
analyzer = new Analyzer(context.getEnv(), context);
+ if (prepareStmt != null) {
Review Comment:
add comment
##########
gensrc/proto/internal_service.proto:
##########
@@ -232,6 +232,38 @@ message PFetchDataResult {
optional bool empty_batch = 6;
};
+message KeyTuple {
+ repeated string key_column_rep = 1;
+}
+
+message UUID {
+ required int64 uuid_high = 1;
+ required int64 uuid_low = 2;
+}
+
+// We use thrift definition for some structure, since TExpr,
+// list<Exprs.TExpr>, Descriptors.TDescriptorTable are all thrift format.
+// Modify them to protobuf is a redundant work.
+message PTabletKeyLookupRequest {
+ required int64 tablet_id = 1;
+ repeated KeyTuple key_tuples = 2;
+
+ // reusable structures
+ // serilized from Descriptors.TDescriptorTable
+ optional UUID uuid = 3;
+ optional bytes desc_tbl = 4;
+ // serilized from TExprList
+ optional bytes output_expr = 5;
Review Comment:
Is output_expr for two purpose, expr for produce output and type of output columns?
##########
fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java:
##########
@@ -0,0 +1,221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import org.apache.doris.analysis.DescriptorTable;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.common.Status;
+import org.apache.doris.proto.InternalService;
+import org.apache.doris.proto.InternalService.KeyTuple;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.rpc.RpcException;
+import org.apache.doris.thrift.TExpr;
+import org.apache.doris.thrift.TExprList;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TResultBatch;
+import org.apache.doris.thrift.TStatusCode;
+
+import com.google.protobuf.ByteString;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TDeserializer;
+import org.apache.thrift.TException;
+import org.apache.thrift.TSerializer;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class PointQueryExec {
+ private static final Logger LOG = LogManager.getLogger(PointQueryExec.class);
+ private TNetworkAddress address;
+ // SlotRef sorted by column id
+ private Map<SlotRef, Expr> equalPredicats;
+ // ByteString serialized for prepared statement
+ private ByteString serializedDescTable;
+ private ByteString serializedOutputExpr;
+ private ArrayList<Expr> outputExprs;
+ private DescriptorTable descriptorTable;
+ private long tabletID = 0;
+ private long timeoutMs = 1000; // default 1s
+
+ private boolean isCancel = false;
+ private boolean isBinaryProtocol = false;
+ private long backendID;
+ // For parepared statement cached structure,
+ // there are some pre caculated structure in Backend TabletFetch service
+ // using this ID to find for this prepared statement
+ private UUID cacheID;
+
+ public PointQueryExec(Map<SlotRef, Expr> equalPredicats, DescriptorTable descTable,
+ ArrayList<Expr> outputExprs) {
+ this.equalPredicats = equalPredicats;
+ this.descriptorTable = descTable;
+ this.outputExprs = outputExprs;
+ }
+
+ void setAddressAndBackendID(TNetworkAddress addr, long backendID) {
+ this.address = addr;
+ this.backendID = backendID;
+ }
+
+ public void setSerializedDescTable(ByteString serializedDescTable) {
+ this.serializedDescTable = serializedDescTable;
+ }
+
+ public void setSerializedOutputExpr(ByteString serializedOutputExpr) {
+ this.serializedOutputExpr = serializedOutputExpr;
+ }
+
+ public void setCacheID(UUID cacheID) {
+ this.cacheID = cacheID;
+ }
+
+ public void setTabletId(long tabletID) {
+ this.tabletID = tabletID;
+ }
+
+ public void setTimeout(long timeoutMs) {
+ this.timeoutMs = timeoutMs;
+ }
+
+ public void setBinaryProtocol(boolean isBinaryProtocol) {
+ this.isBinaryProtocol = isBinaryProtocol;
+ }
+
+ void addKeyTuples(
+ InternalService.PTabletKeyLookupRequest.Builder requestBuilder) {
+ // TODO handle IN predicates
+ KeyTuple.Builder kBuilder = KeyTuple.newBuilder();
+ for (Expr expr : equalPredicats.values()) {
+ LiteralExpr lexpr = (LiteralExpr) expr;
+ kBuilder.addKeyColumnRep(lexpr.getStringValue());
+ }
+ requestBuilder.addKeyTuples(kBuilder);
+ }
+
+ public RowBatch getNext(Status status) throws TException {
+ long timeoutTs = System.currentTimeMillis() + timeoutMs;
+ RowBatch rowBatch = new RowBatch();
+ InternalService.PTabletKeyLookupResponse pResult = null;
+ try {
+ if (serializedDescTable == null) {
Review Comment:
Is serialization repeteated for every getNext call?
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/PlaceHolderExpr.java:
##########
@@ -0,0 +1,180 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.NotImplementedException;
+import org.apache.doris.thrift.TExprNode;
+
+import com.google.common.base.Preconditions;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+// PlaceHolderExpr is a reference class point to real LiteralExpr
+public class PlaceHolderExpr extends LiteralExpr {
+ private static final Logger LOG = LogManager.getLogger(LiteralExpr.class);
+ private LiteralExpr lExpr;
+ int mysqlTypeCode = -1;
+
+ public PlaceHolderExpr() {
+
+ }
+
+ public void setTypeCode(int mysqlTypeCode) {
+ this.mysqlTypeCode = mysqlTypeCode;
+ }
+
+ protected PlaceHolderExpr(LiteralExpr literal) {
+ this.lExpr = literal;
+ }
+
+ protected PlaceHolderExpr(PlaceHolderExpr other) {
+ this.lExpr = other.lExpr;
+ }
+
+ public void setLiteral(LiteralExpr literal) {
+ this.lExpr = literal;
+ this.type = literal.getType();
+ }
+
+ public LiteralExpr createLiteralFromType() throws AnalysisException {
+ Preconditions.checkState(mysqlTypeCode > 0);
+ return LiteralExpr.getLiteralByMysqlType(mysqlTypeCode);
+ }
+
+ public static PlaceHolderExpr create(String value, Type type) throws AnalysisException {
+ Preconditions.checkArgument(!type.equals(Type.INVALID));
+ return new PlaceHolderExpr(LiteralExpr.create(value, type));
+ }
+
+ @Override
+ protected void toThrift(TExprNode msg) {
+ lExpr.toThrift(msg);
+ }
+
+ /*
+ * return real value
+ */
+ public Object getRealValue() {
+ // implemented: TINYINT/SMALLINT/INT/BIGINT/LARGEINT/DATE/DATETIME/CHAR/VARCHAR/BOOLEAN
+ Preconditions.checkState(false, "not implement this in derived class. " + this.type.toSql());
Review Comment:
always false?
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1648,10 +1721,37 @@ private void sendMetaData(ResultSetMetaData metaData) throws IOException {
context.getMysqlChannel().sendOnePacket(serializer.toByteBuffer());
}
+ private void sendStmtPrepareOK() throws IOException {
+ // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_prepare.html#sect_protocol_com_stmt_prepare_response
+ serializer.reset();
+ // 0x00 OK
+ serializer.writeInt1(0);
+ // statement_id
+ serializer.writeInt4(Integer.valueOf(prepareStmt.getName()));
+ // num_columns
+ int numColumns = 0;
+ serializer.writeInt2(numColumns);
+ // num_params
+ int numParams = prepareStmt.getColLabelsOfPlaceHolders().size();
+ serializer.writeInt2(numParams);
+ // reserved_1
+ // serializer.writeInt1(0);
Review Comment:
useless code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1080792839
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: calling a private constructor of class 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:383:** declared private here
```cpp
TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
Review Comment:
warning: calling a private constructor of class 'doris::SlotDescriptor' [clang-diagnostic-error]
```cpp
SlotDescriptor* slot = new SlotDescriptor(tslot1);
^
```
**be/src/runtime/descriptors.h:160:** declared private here
```cpp
SlotDescriptor(const TSlotDescriptor& tdesc);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: calling a private constructor of class 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:383:** declared private here
```cpp
TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ for (auto t : cols) {
+ TSlotDescriptor tslot;
+ tslot.__set_colName(std::get<0>(t));
+ if (std::get<3>(t) == TYPE_DECIMAL128I) {
+ TypeDescriptor type_desc(std::get<3>(t));
+ type_desc.precision = 27;
+ type_desc.scale = 9;
+ tslot.__set_slotType(type_desc.to_thrift());
+ } else {
+ TypeDescriptor type_desc(std::get<3>(t));
+ tslot.__set_slotType(type_desc.to_thrift());
+ }
+ tslot.__set_col_unique_id(std::get<2>(t));
+ SlotDescriptor* slot = new SlotDescriptor(tslot);
+ read_desc.add_slot(slot);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot);
^
```
**be/src/runtime/descriptors.h:387:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot2);
^
```
**be/src/runtime/descriptors.h:387:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot);
^
```
**be/src/runtime/descriptors.h:387:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
Review Comment:
warning: calling a private constructor of class 'doris::SlotDescriptor' [clang-diagnostic-error]
```cpp
SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
^
```
**be/src/runtime/descriptors.h:160:** declared private here
```cpp
SlotDescriptor(const TSlotDescriptor& tdesc);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ for (auto t : cols) {
+ TSlotDescriptor tslot;
+ tslot.__set_colName(std::get<0>(t));
+ if (std::get<3>(t) == TYPE_DECIMAL128I) {
+ TypeDescriptor type_desc(std::get<3>(t));
+ type_desc.precision = 27;
+ type_desc.scale = 9;
+ tslot.__set_slotType(type_desc.to_thrift());
+ } else {
+ TypeDescriptor type_desc(std::get<3>(t));
+ tslot.__set_slotType(type_desc.to_thrift());
+ }
+ tslot.__set_col_unique_id(std::get<2>(t));
+ SlotDescriptor* slot = new SlotDescriptor(tslot);
Review Comment:
warning: calling a private constructor of class 'doris::SlotDescriptor' [clang-diagnostic-error]
```cpp
SlotDescriptor* slot = new SlotDescriptor(tslot);
^
```
**be/src/runtime/descriptors.h:160:** declared private here
```cpp
SlotDescriptor(const TSlotDescriptor& tdesc);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1065315583
##########
be/src/runtime/descriptors.h:
##########
@@ -359,9 +372,12 @@
// Provide quick way to check if there are variable length slots.
// True if _string_slots or _collection_slots have entries.
bool _has_varlen_slots;
+ bool _own_slots = false;
- TupleDescriptor(const TTupleDescriptor& tdesc);
- TupleDescriptor(const PTupleDescriptor& tdesc);
+ TupleDescriptor(const TTupleDescriptor& tdesc, bool own_slot = false);
+ TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
+ TupleDescriptor(TupleDescriptor&&) = delete;
+ void operator=(const TupleDescriptor&) = delete;
Review Comment:
warning: deleted member function should be public [modernize-use-equals-delete]
```cpp
void operator=(const TupleDescriptor&) = delete;
^
```
##########
be/src/runtime/descriptors.h:
##########
@@ -359,9 +372,12 @@ class TupleDescriptor {
// Provide quick way to check if there are variable length slots.
// True if _string_slots or _collection_slots have entries.
bool _has_varlen_slots;
+ bool _own_slots = false;
- TupleDescriptor(const TTupleDescriptor& tdesc);
- TupleDescriptor(const PTupleDescriptor& tdesc);
+ TupleDescriptor(const TTupleDescriptor& tdesc, bool own_slot = false);
+ TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
+ TupleDescriptor(TupleDescriptor&&) = delete;
Review Comment:
warning: deleted member function should be public [modernize-use-equals-delete]
```cpp
TupleDescriptor(TupleDescriptor&&) = delete;
^
```
##########
be/src/util/jsonb_writer.h:
##########
@@ -246,6 +257,18 @@ class JsonbWriterT {
return 0;
}
+ uint32_t writeFloat(float v) {
+ if ((first_ && stack_.empty()) || (!stack_.empty() && verifyValueState())) {
+ if (!writeFirstHeader()) return 0;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (!writeFirstHeader()) { return 0;
}
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: calling a private constructor of class 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:377:** declared private here
```cpp
TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot);
^
```
**be/src/runtime/descriptors.h:381:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ for (auto t : cols) {
+ TSlotDescriptor tslot;
+ tslot.__set_colName(std::get<0>(t));
+ if (std::get<3>(t) == TYPE_DECIMAL128I) {
+ TypeDescriptor type_desc(std::get<3>(t));
+ type_desc.precision = 27;
+ type_desc.scale = 9;
+ tslot.__set_slotType(type_desc.to_thrift());
+ } else {
+ TypeDescriptor type_desc(std::get<3>(t));
+ tslot.__set_slotType(type_desc.to_thrift());
+ }
+ tslot.__set_col_unique_id(std::get<2>(t));
+ SlotDescriptor* slot = new SlotDescriptor(tslot);
+ read_desc.add_slot(slot);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot);
^
```
**be/src/runtime/descriptors.h:381:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128I},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128I},
+ {"k6", OLAP_FIELD_TYPE_INT, 6, TYPE_INT},
+ {"k9", OLAP_FIELD_TYPE_DATEV2, 9, TYPE_DATEV2}};
+ for (auto t : cols) {
+ TabletColumn c;
+ c.set_name(std::get<0>(t));
+ c.set_type(std::get<1>(t));
+ c.set_unique_id(std::get<2>(t));
+ schema.append_column(c);
+ }
+ // int
+ {
+ auto vec = vectorized::ColumnVector<Int32>::create();
+ auto& data = vec->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ data.push_back(i);
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(vec->get_ptr(), data_type, "test_int");
+ block.insert(type_and_name);
+ }
+ // string
+ {
+ auto strcol = vectorized::ColumnString::create();
+ for (int i = 0; i < 1024; ++i) {
+ std::string is = std::to_string(i);
+ strcol->insert_data(is.c_str(), is.size());
+ }
+ vectorized::DataTypePtr data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::ColumnWithTypeAndName type_and_name(strcol->get_ptr(), data_type,
+ "test_string");
+ block.insert(type_and_name);
+ }
+ // decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ auto decimal_column = decimal_data_type->create_column();
+ auto& data = ((vectorized::ColumnDecimal<vectorized::Decimal<vectorized::Int128>>*)
+ decimal_column.get())
+ ->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ __int128_t value = i * pow(10, 9) + i * pow(10, 8);
+ data.push_back(value);
+ }
+ vectorized::ColumnWithTypeAndName type_and_name(decimal_column->get_ptr(),
+ decimal_data_type, "test_decimal");
+ block.insert(type_and_name);
+ }
+ // nullable string
+ {
+ vectorized::DataTypePtr string_data_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(string_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(nullable_column->get_ptr(),
+ nullable_data_type, "test_nullable");
+ block.insert(type_and_name);
+ }
+ // nullable decimal
+ {
+ vectorized::DataTypePtr decimal_data_type(doris::vectorized::create_decimal(27, 9, true));
+ vectorized::DataTypePtr nullable_data_type(
+ std::make_shared<vectorized::DataTypeNullable>(decimal_data_type));
+ auto nullable_column = nullable_data_type->create_column();
+ ((vectorized::ColumnNullable*)nullable_column.get())->insert_null_elements(1024);
+ vectorized::ColumnWithTypeAndName type_and_name(
+ nullable_column->get_ptr(), nullable_data_type, "test_nullable_decimal");
+ block.insert(type_and_name);
+ }
+ // int with 1024 batch size
+ {
+ auto column_vector_int32 = vectorized::ColumnVector<Int32>::create();
+ auto column_nullable_vector = vectorized::make_nullable(std::move(column_vector_int32));
+ auto mutable_nullable_vector = std::move(*column_nullable_vector).mutate();
+ for (int i = 0; i < 1024; i++) {
+ mutable_nullable_vector->insert(vectorized::cast_to_nearest_field_type(i));
+ }
+ auto data_type = vectorized::make_nullable(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::ColumnWithTypeAndName type_and_name(mutable_nullable_vector->get_ptr(),
+ data_type, "test_nullable_int32");
+ block.insert(type_and_name);
+ }
+ // fill with datev2
+ {
+ auto column_vector_date_v2 = vectorized::ColumnVector<vectorized::UInt32>::create();
+ auto& date_v2_data = column_vector_date_v2->get_data();
+ for (int i = 0; i < 1024; ++i) {
+ vectorized::DateV2Value<doris::vectorized::DateV2ValueType> value;
+ value.from_date((uint32_t)((2022 << 9) | (6 << 5) | 6));
+ date_v2_data.push_back(*reinterpret_cast<vectorized::UInt32*>(&value));
+ }
+ vectorized::DataTypePtr date_v2_type(std::make_shared<vectorized::DataTypeDateV2>());
+ vectorized::ColumnWithTypeAndName test_date_v2(column_vector_date_v2->get_ptr(),
+ date_v2_type, "test_datev2");
+ block.insert(test_date_v2);
+ }
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: calling a private constructor of class 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:377:** declared private here
```cpp
TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "gen_cpp/descriptors.pb.h"
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ TSlotDescriptor tslot1;
+ tslot1.__set_colName("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ tslot1.__set_slotType(type_desc.to_thrift());
+ tslot1.__set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(tslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ TSlotDescriptor tslot2;
+ tslot2.__set_colName("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ tslot2.__set_slotType(type_desc2.to_thrift());
+ tslot2.__set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(tslot2);
+ read_desc.add_slot(slot2);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot2);
^
```
**be/src/runtime/descriptors.h:381:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071703770
##########
fe/fe-core/src/main/cup/sql_parser.cup:
##########
@@ -6113,6 +6160,8 @@ literal ::=
{: RESULT = new BoolLiteral(false); :}
| KW_NULL
{: RESULT = new NullLiteral(); :}
+ | PLACEHOLDER
Review Comment:
represend the placeholder marker `?` , eg. `select * from tbl where a = ?`
##########
gensrc/proto/internal_service.proto:
##########
@@ -232,6 +232,38 @@ message PFetchDataResult {
optional bool empty_batch = 6;
};
+message KeyTuple {
+ repeated string key_column_rep = 1;
+}
+
+message UUID {
+ required int64 uuid_high = 1;
+ required int64 uuid_low = 2;
+}
+
+// We use thrift definition for some structure, since TExpr,
+// list<Exprs.TExpr>, Descriptors.TDescriptorTable are all thrift format.
+// Modify them to protobuf is a redundant work.
+message PTabletKeyLookupRequest {
+ required int64 tablet_id = 1;
+ repeated KeyTuple key_tuples = 2;
+
+ // reusable structures
+ // serilized from Descriptors.TDescriptorTable
+ optional UUID uuid = 3;
+ optional bytes desc_tbl = 4;
+ // serilized from TExprList
+ optional bytes output_expr = 5;
+ // return binary mysql row format if true
+ optional bool is_binary_row = 6;
+}
+
+message PTabletKeyLookupResponse {
+ required PStatus status = 1;
+ optional bytes row_batch = 5;
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071704618
##########
gensrc/proto/internal_service.proto:
##########
@@ -232,6 +232,38 @@ message PFetchDataResult {
optional bool empty_batch = 6;
};
+message KeyTuple {
+ repeated string key_column_rep = 1;
+}
+
+message UUID {
+ required int64 uuid_high = 1;
+ required int64 uuid_low = 2;
+}
+
+// We use thrift definition for some structure, since TExpr,
+// list<Exprs.TExpr>, Descriptors.TDescriptorTable are all thrift format.
+// Modify them to protobuf is a redundant work.
+message PTabletKeyLookupRequest {
+ required int64 tablet_id = 1;
+ repeated KeyTuple key_tuples = 2;
+
+ // reusable structures
+ // serilized from Descriptors.TDescriptorTable
+ optional UUID uuid = 3;
+ optional bytes desc_tbl = 4;
Review Comment:
because, desc_tbl and output_expr only exist thrfit format， but this rpc interface is protobuf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071704992
##########
gensrc/proto/internal_service.proto:
##########
@@ -232,6 +232,38 @@ message PFetchDataResult {
optional bool empty_batch = 6;
};
+message KeyTuple {
+ repeated string key_column_rep = 1;
+}
+
+message UUID {
+ required int64 uuid_high = 1;
+ required int64 uuid_low = 2;
+}
+
+// We use thrift definition for some structure, since TExpr,
+// list<Exprs.TExpr>, Descriptors.TDescriptorTable are all thrift format.
+// Modify them to protobuf is a redundant work.
+message PTabletKeyLookupRequest {
+ required int64 tablet_id = 1;
+ repeated KeyTuple key_tuples = 2;
+
+ // reusable structures
+ // serilized from Descriptors.TDescriptorTable
+ optional UUID uuid = 3;
+ optional bytes desc_tbl = 4;
+ // serilized from TExprList
+ optional bytes output_expr = 5;
Review Comment:
yes, used in VMysqlResultWriter for output data like `selec *` or `select func(a)`...
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -1648,10 +1721,37 @@ private void sendMetaData(ResultSetMetaData metaData) throws IOException {
context.getMysqlChannel().sendOnePacket(serializer.toByteBuffer());
}
+ private void sendStmtPrepareOK() throws IOException {
+ // https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_stmt_prepare.html#sect_protocol_com_stmt_prepare_response
+ serializer.reset();
+ // 0x00 OK
+ serializer.writeInt1(0);
+ // statement_id
+ serializer.writeInt4(Integer.valueOf(prepareStmt.getName()));
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071705397
##########
fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java:
##########
@@ -0,0 +1,221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import org.apache.doris.analysis.DescriptorTable;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.common.Status;
+import org.apache.doris.proto.InternalService;
+import org.apache.doris.proto.InternalService.KeyTuple;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.rpc.RpcException;
+import org.apache.doris.thrift.TExpr;
+import org.apache.doris.thrift.TExprList;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TResultBatch;
+import org.apache.doris.thrift.TStatusCode;
+
+import com.google.protobuf.ByteString;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TDeserializer;
+import org.apache.thrift.TException;
+import org.apache.thrift.TSerializer;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class PointQueryExec {
+ private static final Logger LOG = LogManager.getLogger(PointQueryExec.class);
+ private TNetworkAddress address;
+ // SlotRef sorted by column id
+ private Map<SlotRef, Expr> equalPredicats;
+ // ByteString serialized for prepared statement
+ private ByteString serializedDescTable;
+ private ByteString serializedOutputExpr;
+ private ArrayList<Expr> outputExprs;
+ private DescriptorTable descriptorTable;
+ private long tabletID = 0;
+ private long timeoutMs = 1000; // default 1s
+
+ private boolean isCancel = false;
+ private boolean isBinaryProtocol = false;
+ private long backendID;
+ // For parepared statement cached structure,
+ // there are some pre caculated structure in Backend TabletFetch service
+ // using this ID to find for this prepared statement
+ private UUID cacheID;
+
+ public PointQueryExec(Map<SlotRef, Expr> equalPredicats, DescriptorTable descTable,
+ ArrayList<Expr> outputExprs) {
+ this.equalPredicats = equalPredicats;
+ this.descriptorTable = descTable;
+ this.outputExprs = outputExprs;
+ }
+
+ void setAddressAndBackendID(TNetworkAddress addr, long backendID) {
+ this.address = addr;
+ this.backendID = backendID;
+ }
+
+ public void setSerializedDescTable(ByteString serializedDescTable) {
+ this.serializedDescTable = serializedDescTable;
+ }
+
+ public void setSerializedOutputExpr(ByteString serializedOutputExpr) {
+ this.serializedOutputExpr = serializedOutputExpr;
+ }
+
+ public void setCacheID(UUID cacheID) {
+ this.cacheID = cacheID;
+ }
+
+ public void setTabletId(long tabletID) {
+ this.tabletID = tabletID;
+ }
+
+ public void setTimeout(long timeoutMs) {
+ this.timeoutMs = timeoutMs;
+ }
+
+ public void setBinaryProtocol(boolean isBinaryProtocol) {
+ this.isBinaryProtocol = isBinaryProtocol;
+ }
+
+ void addKeyTuples(
+ InternalService.PTabletKeyLookupRequest.Builder requestBuilder) {
+ // TODO handle IN predicates
+ KeyTuple.Builder kBuilder = KeyTuple.newBuilder();
+ for (Expr expr : equalPredicats.values()) {
+ LiteralExpr lexpr = (LiteralExpr) expr;
+ kBuilder.addKeyColumnRep(lexpr.getStringValue());
+ }
+ requestBuilder.addKeyTuples(kBuilder);
+ }
+
+ public RowBatch getNext(Status status) throws TException {
+ long timeoutTs = System.currentTimeMillis() + timeoutMs;
+ RowBatch rowBatch = new RowBatch();
+ InternalService.PTabletKeyLookupResponse pResult = null;
+ try {
+ if (serializedDescTable == null) {
Review Comment:
No,for prepared statement it's only serialized once and will be reused later on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1065298618
##########
be/src/util/jsonb_writer.h:
##########
@@ -234,6 +234,17 @@ class JsonbWriterT {
return 0;
}
+ uint32_t writeInt128(__int128_t v) {
+ if (!stack_.empty() && verifyValueState()) {
Review Comment:
done
##########
fe/fe-core/src/main/java/org/apache/doris/qe/PointQueryExec.java:
##########
@@ -0,0 +1,221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import org.apache.doris.analysis.DescriptorTable;
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.common.Status;
+import org.apache.doris.proto.InternalService;
+import org.apache.doris.proto.InternalService.KeyTuple;
+import org.apache.doris.rpc.BackendServiceProxy;
+import org.apache.doris.rpc.RpcException;
+import org.apache.doris.thrift.TExpr;
+import org.apache.doris.thrift.TExprList;
+import org.apache.doris.thrift.TNetworkAddress;
+import org.apache.doris.thrift.TResultBatch;
+import org.apache.doris.thrift.TStatusCode;
+
+import com.google.protobuf.ByteString;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.thrift.TDeserializer;
+import org.apache.thrift.TException;
+import org.apache.thrift.TSerializer;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class PointQueryExec {
+ private static final Logger LOG = LogManager.getLogger(PointQueryExec.class);
+ private TNetworkAddress address;
+ // SlotRef sorted by column id
+ private Map<SlotRef, Expr> equalPredicats;
+ // ByteString serialized for prepared statement
+ private ByteString serializedDescTable;
+ private ByteString serializedOutputExpr;
+ private ArrayList<Expr> outputExprs;
+ private DescriptorTable descriptorTable;
+ private long tabletID = 0;
+ private long timeoutMs = 1000; // default 1s
+
+ private boolean isCancel = false;
+ private boolean isBinaryProtocol = false;
+ private long backendID;
+ // For parepared statement cached structure,
+ // there are some pre caculated structure in Backend TabletFetch service
+ // using this ID to find for this prepared statement
+ private UUID cacheID;
+
+ public PointQueryExec(Map<SlotRef, Expr> equalPredicats, DescriptorTable descTable,
+ ArrayList<Expr> outputExprs) {
+ this.equalPredicats = equalPredicats;
+ this.descriptorTable = descTable;
+ this.outputExprs = outputExprs;
+ }
+
+ void setAddressAndBackendID(TNetworkAddress addr, long backendID) {
+ this.address = addr;
+ this.backendID = backendID;
+ }
+
+ public void setSerializedDescTable(ByteString serializedDescTable) {
+ this.serializedDescTable = serializedDescTable;
+ }
+
+ public void setSerializedOutputExpr(ByteString serializedOutputExpr) {
+ this.serializedOutputExpr = serializedOutputExpr;
+ }
+
+ public void setCacheID(UUID cacheID) {
+ this.cacheID = cacheID;
+ }
+
+ public void setTabletId(long tabletID) {
+ this.tabletID = tabletID;
+ }
+
+ public void setTimeout(long timeoutMs) {
+ this.timeoutMs = timeoutMs;
+ }
+
+ public void setBinaryProtocol(boolean isBinaryProtocol) {
+ this.isBinaryProtocol = isBinaryProtocol;
+ }
+
+ void addKeyTuples(
+ InternalService.PTabletKeyLookupRequest.Builder requestBuilder) {
+ // TODO handle IN predicates
+ KeyTuple.Builder kBuilder = KeyTuple.newBuilder();
+ for (Expr expr : equalPredicats.values()) {
+ LiteralExpr lexpr = (LiteralExpr) expr;
+ kBuilder.addKeyColumnRep(lexpr.getStringValue());
+ }
+ requestBuilder.addKeyTuples(kBuilder);
+ }
+
+ public RowBatch getNext(Status status) throws TException {
+ long timeoutTs = System.currentTimeMillis() + timeoutMs;
+ RowBatch rowBatch = new RowBatch();
+ InternalService.PTabletKeyLookupResponse pResult = null;
+ try {
+ if (serializedDescTable == null) {
+ serializedDescTable = ByteString.copyFrom(
+ new TSerializer().serialize(descriptorTable.toThrift()));
+ }
+ if (serializedOutputExpr == null) {
+ List<TExpr> exprs = new ArrayList<>();
+ for (Expr expr : outputExprs) {
+ exprs.add(expr.treeToThrift());
+ }
+ TExprList exprList = new TExprList(exprs);
+ serializedOutputExpr = ByteString.copyFrom(
+ new TSerializer().serialize(exprList));
+ }
+
+ InternalService.PTabletKeyLookupRequest.Builder requestBuilder
+ = InternalService.PTabletKeyLookupRequest.newBuilder()
+ .setTabletId(tabletID)
+ .setDescTbl(serializedDescTable)
+ .setOutputExpr(serializedOutputExpr)
+ .setIsBinaryRow(isBinaryProtocol);
+ if (cacheID != null) {
+ InternalService.UUID.Builder uuidBuilder = InternalService.UUID.newBuilder();
+ uuidBuilder.setUuidHigh(cacheID.getMostSignificantBits());
+ uuidBuilder.setUuidLow(cacheID.getLeastSignificantBits());
+ requestBuilder.setUuid(uuidBuilder);
+ }
+ addKeyTuples(requestBuilder);
+ InternalService.PTabletKeyLookupRequest request = requestBuilder.build();
+ Future<InternalService.PTabletKeyLookupResponse> futureResponse =
+ BackendServiceProxy.getInstance().fetchTabletDataAsync(address, request);
+ while (pResult == null) {
+ long currentTs = System.currentTimeMillis();
+ if (currentTs >= timeoutTs) {
+ LOG.warn("fetch result timeout {}", address);
+ status.setStatus("query timeout");
+ return null;
Review Comment:
need retry logic
##########
be/src/util/mysql_global.h:
##########
@@ -26,13 +26,16 @@ typedef unsigned char uchar;
#define int1store(T, A) *((uint8_t*)(T)) = (uint8_t)(A)
#define int2store(T, A) *((uint16_t*)(T)) = (uint16_t)(A)
+#define int4store(T, A) *((uint32_t*)(T)) = (uint32_t)(A)
#define int3store(T, A) \
do { \
*(T) = (uchar)((A)); \
*(T + 1) = (uchar)(((uint32_t)(A) >> 8)); \
*(T + 2) = (uchar)(((A) >> 16)); \
} while (0)
#define int8store(T, A) *((int64_t*)(T)) = (uint64_t)(A)
+#define float4store(T, A) memcpy(T, (&A), sizeof(float))
Review Comment:
done
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (type == TYPE_HLL) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (is_string_type(type)) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->insert_data(blob->getBlob(), blob->getBlobLen());
+ } else {
+ switch (type) {
+ case TYPE_BOOLEAN: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_TINYINT: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_SMALLINT: {
+ assert(slot_value->isInt16());
+ dst->insert(static_cast<JsonbInt16Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_INT: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_BIGINT: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_LARGEINT: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_FLOAT:
Review Comment:
done
##########
be/src/service/internal_service.cpp:
##########
@@ -391,6 +393,25 @@ void PInternalServiceImpl::fetch_table_schema(google::protobuf::RpcController* c
st.to_protobuf(result->mutable_status());
}
+Status PInternalServiceImpl::_tablet_fetch_data(const PTabletKeyLookupRequest* request,
+ PTabletKeyLookupResponse* response) {
+ TabletLookupMetric lookup_util;
+ RETURN_IF_ERROR(lookup_util.init(request, response));
+ RETURN_IF_ERROR(lookup_util.lookup_up());
+ LOG(INFO) << lookup_util.print_profile();
Review Comment:
INFO is suitable, but it maybe too frequent, so I changed it to LOG_EVERY_N
##########
be/src/service/tablet_lookup_metric.h:
##########
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "butil/containers/doubly_buffered_data.h"
+#include "common/status.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gutil/int128.h"
+#include "olap/tablet.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+
+// For caching point lookup pre allocted blocks and exprs
+class Reusable {
+public:
+ ~Reusable();
+
+ bool is_expired(int64_t ttl_ms) const {
+ return butil::gettimeofday_ms() - _create_timestamp > ttl_ms;
+ }
+
+ Status init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size = 1);
+
+ RuntimeState* runtime_state() { return _runtime_state.get(); }
+
+ std::unique_ptr<vectorized::Block> get_block();
+
+ // do not touch block after returned
+ void return_block(std::unique_ptr<vectorized::Block>& block);
+
+ TupleDescriptor* tuple_desc() { return _desc_tbl->get_tuple_descriptor(0); }
+
+ const std::vector<vectorized::VExprContext*>& output_exprs() { return _output_exprs_ctxs; }
+
+private:
+ // caching TupleDescriptor, output_expr, etc...
+ std::unique_ptr<RuntimeState> _runtime_state;
+ DescriptorTbl* _desc_tbl;
+ std::mutex _block_mutex;
+ // prevent from allocte too many tmp blocks
+ std::vector<std::unique_ptr<vectorized::Block>> _block_pool;
+ std::vector<vectorized::VExprContext*> _output_exprs_ctxs;
+ int64_t _create_timestamp = 0;
+};
+
+// A cache used for prepare stmt.
+// One connection per stmt perf uuid
+// Use DoublyBufferedData to wrap Cache for performance and thread safe,
+// since it's not barely modified
+class LookupCache {
+public:
+ // uuid to reusable
+ using Cache = phmap::flat_hash_map<uint128, std::shared_ptr<Reusable>>;
+ using CacheIter = Cache::iterator;
+
+ LookupCache() = default;
+ static LookupCache& instance() {
+ static LookupCache ins;
+ return ins;
+ }
+
+ void add(uint128 cache_id, std::shared_ptr<Reusable> item) {
+ assert(item != nullptr);
+ _double_buffer_cache.Modify(update_cache, std::make_pair(cache_id, item));
+ }
+
+ // find an item, return null if not exist
+ std::shared_ptr<Reusable> get(uint128 cache_id) {
+ butil::DoublyBufferedData<Cache>::ScopedPtr s;
+ if (_double_buffer_cache.Read(&s) != 0) {
+ LOG(WARNING) << "failed to get cache from double buffer data";
+ return nullptr;
+ }
+ auto it = s->find(cache_id);
+ if (it != s->end()) {
+ return it->second;
+ }
+ return nullptr;
+ }
+
+private:
+ butil::DoublyBufferedData<Cache> _double_buffer_cache;
+ // 30 seconds for expiring an item
+ int32_t _expir_seconds = config::tablet_lookup_cache_clean_interval;
+
+ static size_t update_cache(Cache& old_cache,
+ const std::pair<uint128, std::shared_ptr<Reusable>>& p) {
+ old_cache.emplace(p);
+ return 1;
+ }
+
+ static size_t remove_items(Cache& old_cache, const std::vector<uint128>& keys) {
+ for (size_t i = 0; i < keys.size(); ++i) {
+ old_cache.erase(keys[i]);
+ }
+ return 1;
+ }
+
+ // Called from StorageEngine::_start_clean_lookup_cache
+ friend class StorageEngine;
+ void prune() {
+ std::vector<uint128> expired_keys;
+ {
+ butil::DoublyBufferedData<Cache>::ScopedPtr s;
+ if (_double_buffer_cache.Read(&s) != 0) {
+ return;
+ }
+ for (auto it = s->begin(); it != s->end(); ++it) {
+ if (it->second->is_expired(_expir_seconds * 1000)) {
+ expired_keys.push_back(it->first);
+ }
+ }
+ }
+
+ _double_buffer_cache.Modify(remove_items, expired_keys);
+ LOG(INFO) << "prune lookup cache, total " << expired_keys.size() << " expired items";
+ }
+};
+
+struct Metrics {
+ Metrics()
+ : init_ns(TUnit::TIME_NS),
+ init_key_ns(TUnit::TIME_NS),
+ lookup_key_ns(TUnit::TIME_NS),
+ lookup_data_ns(TUnit::TIME_NS),
+ output_data_ns(TUnit::TIME_NS) {}
+ RuntimeProfile::Counter init_ns;
+ RuntimeProfile::Counter init_key_ns;
+ RuntimeProfile::Counter lookup_key_ns;
+ RuntimeProfile::Counter lookup_data_ns;
+ RuntimeProfile::Counter output_data_ns;
+};
+
+// An util to do tablet lookup
+class TabletLookupMetric {
Review Comment:
changed to `PointQueryExecutor`
##########
be/src/olap/tablet_schema.cpp:
##########
@@ -920,11 +923,25 @@ bool operator==(const TabletSchema& a, const TabletSchema& b) {
if (a._is_in_memory != b._is_in_memory) return false;
if (a._delete_sign_idx != b._delete_sign_idx) return false;
if (a._disable_auto_compaction != b._disable_auto_compaction) return false;
+ if (a._store_row_column != b._store_row_column) return false;
return true;
}
bool operator!=(const TabletSchema& a, const TabletSchema& b) {
return !(a == b);
}
+const TabletColumn& TabletSchema::row_oriented_column() {
+ static TabletColumn source_column(OLAP_FIELD_AGGREGATION_NONE,
Review Comment:
already return const
##########
be/src/util/mysql_row_buffer.cpp:
##########
@@ -290,7 +392,11 @@ int MysqlRowBuffer::push_double(double data) {
return 0;
}
-int MysqlRowBuffer::push_time(double data) {
+template <bool is_binary_format>
+int MysqlRowBuffer<is_binary_format>::push_time(double data) {
+ if constexpr (is_binary_format) {
+ LOG(FATAL) << "not implemented";
Review Comment:
added
##########
be/src/util/jsonb_stream.h:
##########
@@ -126,6 +126,11 @@ class JsonbOutStream : public std::ostream {
size_ += len;
}
+ void write(__int128 l) {
+ // TODO
Review Comment:
this function is not used for now
##########
be/src/service/tablet_lookup_metric.h:
##########
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "butil/containers/doubly_buffered_data.h"
+#include "common/status.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gutil/int128.h"
+#include "olap/tablet.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+
+// For caching point lookup pre allocted blocks and exprs
+class Reusable {
+public:
+ ~Reusable();
+
+ bool is_expired(int64_t ttl_ms) const {
+ return butil::gettimeofday_ms() - _create_timestamp > ttl_ms;
+ }
+
+ Status init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size = 1);
+
+ RuntimeState* runtime_state() { return _runtime_state.get(); }
+
+ std::unique_ptr<vectorized::Block> get_block();
+
+ // do not touch block after returned
+ void return_block(std::unique_ptr<vectorized::Block>& block);
+
+ TupleDescriptor* tuple_desc() { return _desc_tbl->get_tuple_descriptor(0); }
+
+ const std::vector<vectorized::VExprContext*>& output_exprs() { return _output_exprs_ctxs; }
+
+private:
+ // caching TupleDescriptor, output_expr, etc...
+ std::unique_ptr<RuntimeState> _runtime_state;
Review Comment:
it's safe since Reusable is wrapped in std::shared_ptr
##########
be/src/vec/CMakeLists.txt:
##########
@@ -55,6 +55,7 @@ set(VEC_FILES
columns/column_string.cpp
columns/column_vector.cpp
columns/columns_common.cpp
+ jsonb/serialize.cpp
Review Comment:
done
##########
be/src/util/mysql_global.h:
##########
@@ -26,13 +26,16 @@ typedef unsigned char uchar;
#define int1store(T, A) *((uint8_t*)(T)) = (uint8_t)(A)
#define int2store(T, A) *((uint16_t*)(T)) = (uint16_t)(A)
+#define int4store(T, A) *((uint32_t*)(T)) = (uint32_t)(A)
Review Comment:
done
##########
be/src/service/tablet_lookup_metric.cpp:
##########
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "service/tablet_lookup_metric.h"
+
+#include "olap/row_cursor.h"
+#include "olap/storage_engine.h"
+#include "service/internal_service.h"
+#include "util/defer_op.h"
+#include "util/key_util.h"
+#include "util/runtime_profile.h"
+#include "util/thrift_util.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vliteral.h"
+#include "vec/sink/vmysql_result_writer.cpp"
+
+namespace doris {
+
+Reusable::~Reusable() {
+ for (vectorized::VExprContext* ctx : _output_exprs_ctxs) {
+ ctx->close(_runtime_state.get());
+ }
+}
+
+Status Reusable::init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size) {
+ _runtime_state.reset(new RuntimeState());
+ RETURN_IF_ERROR(DescriptorTbl::create(_runtime_state->obj_pool(), t_desc_tbl, &_desc_tbl));
+ _runtime_state->set_desc_tbl(_desc_tbl);
+ _block_pool.resize(block_size);
+ for (int i = 0; i < _block_pool.size(); ++i) {
+ _block_pool[i] = std::make_unique<vectorized::Block>(tuple_desc()->slots(), 10);
+ }
+
+ RETURN_IF_ERROR(vectorized::VExpr::create_expr_trees(_runtime_state->obj_pool(), output_exprs,
+ &_output_exprs_ctxs));
+ RowDescriptor row_desc(tuple_desc(), false);
+ // Prepare the exprs to run.
+ RETURN_IF_ERROR(vectorized::VExpr::prepare(_output_exprs_ctxs, _runtime_state.get(), row_desc));
+ _create_timestamp = butil::gettimeofday_ms();
+ return Status::OK();
+}
+
+std::unique_ptr<vectorized::Block> Reusable::get_block() {
+ std::lock_guard lock(_block_mutex);
+ if (_block_pool.empty()) {
+ return std::make_unique<vectorized::Block>(tuple_desc()->slots(), 4);
Review Comment:
may lead to too many config, so I use magic number here
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (type == TYPE_HLL) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (is_string_type(type)) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->insert_data(blob->getBlob(), blob->getBlobLen());
+ } else {
+ switch (type) {
+ case TYPE_BOOLEAN: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_TINYINT: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_SMALLINT: {
+ assert(slot_value->isInt16());
+ dst->insert(static_cast<JsonbInt16Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_INT: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_BIGINT: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_LARGEINT: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE: {
+ assert(slot_value->isDouble());
+ dst->insert(static_cast<JsonbDoubleVal*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATE: {
+ assert(slot_value->isInt32());
+ int32_t val = static_cast<JsonbInt32Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATETIME: {
+ assert(slot_value->isInt64());
+ int64_t val = static_cast<JsonbInt64Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATEV2: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATETIMEV2: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL32: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL64: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL128I: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ default:
+ LOG(FATAL) << "unknow type " << type;
+ break;
+ }
+ }
+}
+
+// column value -> jsonb
+static void serialize_column(Arena* mem_pool, const TabletColumn& tablet_column,
+ const IColumn* column, const StringRef& data_ref, int row,
+ JsonbWriterT<JsonbOutStream>& jsonb_writer) {
+ jsonb_writer.writeKey(tablet_column.unique_id());
+ if (is_column_null_at(row, column, tablet_column.type(), data_ref)) {
+ jsonb_writer.writeNull();
+ return;
+ }
+ if (tablet_column.is_array_type()) {
+ const char* begin = nullptr;
+ StringRef value = column->serialize_value_into_arena(row, *mem_pool, begin);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(value.data, value.size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_OBJECT) {
+ auto bitmap_value = (BitmapValue*)(data_ref.data);
+ auto size = bitmap_value->getSizeInBytes();
+ // serialize the content of string
+ auto ptr = mem_pool->alloc(size);
+ bitmap_value->write(reinterpret_cast<char*>(ptr));
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_HLL) {
+ auto hll_value = (HyperLogLog*)(data_ref.data);
+ auto size = hll_value->max_serialized_size();
+ auto ptr = reinterpret_cast<char*>(mem_pool->alloc(size));
+ size_t actual_size = hll_value->serialize((uint8_t*)ptr);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), actual_size);
+ jsonb_writer.writeEndBinary();
+ } else if (is_jsonb_blob_type(tablet_column.type())) {
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(data_ref.data), data_ref.size);
+ jsonb_writer.writeEndBinary();
+ } else {
+ switch (tablet_column.type()) {
+ case OLAP_FIELD_TYPE_BOOL: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_TINYINT: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_SMALLINT: {
+ int16_t val = *reinterpret_cast<const int16_t*>(data_ref.data);
+ jsonb_writer.writeInt16(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_INT: {
+ int32_t val = *reinterpret_cast<const int32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_BIGINT: {
+ int64_t val = *reinterpret_cast<const int64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_LARGEINT: {
+ __int128_t val = *reinterpret_cast<const __int128_t*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_FLOAT: {
+ float val = *reinterpret_cast<const float*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DOUBLE: {
+ double val = *reinterpret_cast<const double*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATE: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt32(datetime_cur->to_olap_date());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIME: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt64(datetime_cur->to_olap_datetime());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATEV2: {
+ uint32_t val = *reinterpret_cast<const uint32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIMEV2: {
+ uint64_t val = *reinterpret_cast<const uint64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL32: {
+ Decimal32::NativeType val =
+ *reinterpret_cast<const Decimal32::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL64: {
+ Decimal64::NativeType val =
+ *reinterpret_cast<const Decimal64::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL128I: {
+ Decimal128I::NativeType val =
+ *reinterpret_cast<const Decimal128I::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL:
+ LOG(FATAL) << "OLAP_FIELD_TYPE_DECIMAL not implemented use DecimalV3 instead";
+ break;
+ default:
+ LOG(FATAL) << "unknow type " << tablet_column.type();
+ break;
+ }
+ }
+}
+
+void JsonbSerializeUtil::block_to_jsonb(const TabletSchema& schema, const Block& block,
+ ColumnString& dst, int num_cols) {
+ auto num_rows = block.rows();
+ Arena pool;
+ assert(num_cols <= block.columns());
+ for (int i = 0; i < num_rows; ++i) {
+ JsonbWriterT<JsonbOutStream> jsonb_writer;
Review Comment:
resue JsonbWriterT should call `reset` before reusing it, but `reset` is slower than it's destructor
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
Review Comment:
done
##########
be/src/olap/tablet.cpp:
##########
@@ -1844,6 +1846,103 @@ TabletSchemaSPtr Tablet::get_max_version_schema(std::lock_guard<std::shared_mute
return _max_version_schema;
}
+RowsetSharedPtr Tablet::get_rowset(const RowsetId& rowset_id) {
+ for (auto& version_rowset : _rs_version_map) {
+ if (version_rowset.second->rowset_id() == rowset_id) {
+ return version_rowset.second;
+ }
+ }
+ for (auto& stale_version_rowset : _stale_rs_version_map) {
+ if (stale_version_rowset.second->rowset_id() == rowset_id) {
+ return stale_version_rowset.second;
+ }
+ }
+ return nullptr;
+}
+
+Status Tablet::lookup_row_data(const RowLocation& row_location, const TupleDescriptor* desc,
+ vectorized::Block* block) {
+ // read row data
+ BetaRowsetSharedPtr rowset =
+ std::static_pointer_cast<BetaRowset>(get_rowset(row_location.rowset_id));
+ if (!rowset) {
+ return Status::NotFound(
+ fmt::format("rowset {} not found", row_location.rowset_id.to_string()));
+ }
+
+ const TabletSchemaSPtr tablet_schema = rowset->tablet_schema();
+ SegmentCacheHandle segment_cache;
+ RETURN_IF_ERROR(SegmentLoader::instance()->load_segments(rowset, &segment_cache, true));
+ // find segment
+ auto it = std::find_if(segment_cache.get_segments().begin(), segment_cache.get_segments().end(),
+ [&row_location](const segment_v2::SegmentSharedPtr& seg) {
+ return seg->id() == row_location.segment_id;
+ });
+ if (it == segment_cache.get_segments().end()) {
+ return Status::NotFound(fmt::format("rowset {} 's segemnt not found, seg_id {}",
+ row_location.rowset_id.to_string(),
+ row_location.segment_id));
+ }
+ // read from segment column by column, row by row
+ segment_v2::SegmentSharedPtr segment = *it;
+ size_t row_size = 0;
+ MonotonicStopWatch watch;
+ watch.start();
+ Defer _defer([&]() {
+ LOG_EVERY_N(INFO, 500) << "get a single_row, cost(us):" << watch.elapsed_time() / 1000
+ << ", row_size:" << row_size;
+ });
+ // TODO(lhy) too long, refacor
+ if (tablet_schema->store_row_column()) {
+ // create _source column
+ segment_v2::ColumnIterator* column_iterator = nullptr;
+ RETURN_IF_ERROR(segment->new_row_column_iterator(&column_iterator));
+ std::unique_ptr<segment_v2::ColumnIterator> ptr_guard(column_iterator);
+ segment_v2::ColumnIteratorOptions opt;
+ OlapReaderStatistics stats;
+ opt.file_reader = segment->file_reader().get();
+ opt.stats = &stats;
+ opt.use_page_cache = !config::disable_storage_page_cache;
+ column_iterator->init(opt);
+ // get and parse tuple row
+ vectorized::MutableColumnPtr column_ptr =
+ vectorized::DataTypeFactory::instance()
+ .create_data_type(TabletSchema::row_oriented_column())
+ ->create_column();
+ std::vector<segment_v2::rowid_t> rowids {
+ static_cast<segment_v2::rowid_t>(row_location.row_id)};
+ RETURN_IF_ERROR(column_iterator->read_by_rowids(rowids.data(), 1, column_ptr));
+ assert(column_ptr->size() == 1);
+ auto string_column = static_cast<vectorized::ColumnString*>(column_ptr.get());
+ vectorized::JsonbSerializeUtil::jsonb_to_block(*desc, *string_column, *block);
+ return Status::OK();
+ }
+ // read from segment column by column, row by row
+ for (int x = 0; x < desc->slots().size(); ++x) {
+ int index = tablet_schema->field_index(desc->slots()[x]->col_unique_id());
+ auto column = block->get_by_position(x).column->assume_mutable();
+ // TODO handle real default value
Review Comment:
deleted this logic, only use row store logic now
##########
be/src/runtime/descriptors.h:
##########
@@ -302,6 +307,13 @@ class JdbcTableDescriptor : public TableDescriptor {
class TupleDescriptor {
public:
// virtual ~TupleDescriptor() {}
+ ~TupleDescriptor() {
Review Comment:
added
##########
be/src/olap/delta_writer.cpp:
##########
@@ -440,6 +440,7 @@ void DeltaWriter::_build_current_tablet_schema(int64_t index_id,
ptable_schema_param.indexes(i),
ori_tablet_schema);
}
+
Review Comment:
deleted
##########
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##########
@@ -244,6 +244,7 @@ Status SegmentIterator::_get_row_ranges_by_keys() {
auto row_range = RowRanges::create_single(lower_rowid, upper_rowid);
RowRanges::ranges_union(result_ranges, row_range, &result_ranges);
}
+
Review Comment:
deleted
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (type == TYPE_HLL) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (is_string_type(type)) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->insert_data(blob->getBlob(), blob->getBlobLen());
+ } else {
+ switch (type) {
+ case TYPE_BOOLEAN: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_TINYINT: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_SMALLINT: {
+ assert(slot_value->isInt16());
+ dst->insert(static_cast<JsonbInt16Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_INT: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_BIGINT: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_LARGEINT: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE: {
+ assert(slot_value->isDouble());
+ dst->insert(static_cast<JsonbDoubleVal*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATE: {
+ assert(slot_value->isInt32());
+ int32_t val = static_cast<JsonbInt32Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATETIME: {
+ assert(slot_value->isInt64());
+ int64_t val = static_cast<JsonbInt64Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATEV2: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATETIMEV2: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL32: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL64: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL128I: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ default:
+ LOG(FATAL) << "unknow type " << type;
+ break;
+ }
+ }
+}
+
+// column value -> jsonb
+static void serialize_column(Arena* mem_pool, const TabletColumn& tablet_column,
+ const IColumn* column, const StringRef& data_ref, int row,
+ JsonbWriterT<JsonbOutStream>& jsonb_writer) {
+ jsonb_writer.writeKey(tablet_column.unique_id());
+ if (is_column_null_at(row, column, tablet_column.type(), data_ref)) {
+ jsonb_writer.writeNull();
+ return;
+ }
+ if (tablet_column.is_array_type()) {
+ const char* begin = nullptr;
+ StringRef value = column->serialize_value_into_arena(row, *mem_pool, begin);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(value.data, value.size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_OBJECT) {
+ auto bitmap_value = (BitmapValue*)(data_ref.data);
+ auto size = bitmap_value->getSizeInBytes();
+ // serialize the content of string
+ auto ptr = mem_pool->alloc(size);
+ bitmap_value->write(reinterpret_cast<char*>(ptr));
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_HLL) {
+ auto hll_value = (HyperLogLog*)(data_ref.data);
+ auto size = hll_value->max_serialized_size();
+ auto ptr = reinterpret_cast<char*>(mem_pool->alloc(size));
+ size_t actual_size = hll_value->serialize((uint8_t*)ptr);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), actual_size);
+ jsonb_writer.writeEndBinary();
+ } else if (is_jsonb_blob_type(tablet_column.type())) {
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(data_ref.data), data_ref.size);
+ jsonb_writer.writeEndBinary();
+ } else {
+ switch (tablet_column.type()) {
+ case OLAP_FIELD_TYPE_BOOL: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_TINYINT: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_SMALLINT: {
+ int16_t val = *reinterpret_cast<const int16_t*>(data_ref.data);
+ jsonb_writer.writeInt16(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_INT: {
+ int32_t val = *reinterpret_cast<const int32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_BIGINT: {
+ int64_t val = *reinterpret_cast<const int64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_LARGEINT: {
+ __int128_t val = *reinterpret_cast<const __int128_t*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_FLOAT: {
+ float val = *reinterpret_cast<const float*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DOUBLE: {
+ double val = *reinterpret_cast<const double*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATE: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt32(datetime_cur->to_olap_date());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIME: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt64(datetime_cur->to_olap_datetime());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATEV2: {
+ uint32_t val = *reinterpret_cast<const uint32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIMEV2: {
+ uint64_t val = *reinterpret_cast<const uint64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL32: {
+ Decimal32::NativeType val =
+ *reinterpret_cast<const Decimal32::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL64: {
+ Decimal64::NativeType val =
+ *reinterpret_cast<const Decimal64::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL128I: {
+ Decimal128I::NativeType val =
+ *reinterpret_cast<const Decimal128I::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL:
+ LOG(FATAL) << "OLAP_FIELD_TYPE_DECIMAL not implemented use DecimalV3 instead";
+ break;
+ default:
+ LOG(FATAL) << "unknow type " << tablet_column.type();
+ break;
+ }
+ }
+}
+
+void JsonbSerializeUtil::block_to_jsonb(const TabletSchema& schema, const Block& block,
+ ColumnString& dst, int num_cols) {
+ auto num_rows = block.rows();
+ Arena pool;
+ assert(num_cols <= block.columns());
+ for (int i = 0; i < num_rows; ++i) {
+ JsonbWriterT<JsonbOutStream> jsonb_writer;
+ jsonb_writer.writeStartObject();
+ for (int j = 0; j < num_cols; ++j) {
+ const auto& column = block.get_by_position(j).column;
+ const auto& tablet_column = schema.columns()[j];
+ const auto& data_ref =
+ !tablet_column.is_array_type() ? column->get_data_at(i) : StringRef();
Review Comment:
since ColumnArray is not continous memory for function `get_data_at `
##########
be/src/service/tablet_lookup_metric.cpp:
##########
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "service/tablet_lookup_metric.h"
+
+#include "olap/row_cursor.h"
+#include "olap/storage_engine.h"
+#include "service/internal_service.h"
+#include "util/defer_op.h"
+#include "util/key_util.h"
+#include "util/runtime_profile.h"
+#include "util/thrift_util.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vliteral.h"
+#include "vec/sink/vmysql_result_writer.cpp"
+
+namespace doris {
+
+Reusable::~Reusable() {
+ for (vectorized::VExprContext* ctx : _output_exprs_ctxs) {
+ ctx->close(_runtime_state.get());
+ }
+}
+
+Status Reusable::init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size) {
+ _runtime_state.reset(new RuntimeState());
+ RETURN_IF_ERROR(DescriptorTbl::create(_runtime_state->obj_pool(), t_desc_tbl, &_desc_tbl));
+ _runtime_state->set_desc_tbl(_desc_tbl);
+ _block_pool.resize(block_size);
+ for (int i = 0; i < _block_pool.size(); ++i) {
+ _block_pool[i] = std::make_unique<vectorized::Block>(tuple_desc()->slots(), 10);
+ }
+
+ RETURN_IF_ERROR(vectorized::VExpr::create_expr_trees(_runtime_state->obj_pool(), output_exprs,
+ &_output_exprs_ctxs));
+ RowDescriptor row_desc(tuple_desc(), false);
+ // Prepare the exprs to run.
+ RETURN_IF_ERROR(vectorized::VExpr::prepare(_output_exprs_ctxs, _runtime_state.get(), row_desc));
+ _create_timestamp = butil::gettimeofday_ms();
+ return Status::OK();
+}
+
+std::unique_ptr<vectorized::Block> Reusable::get_block() {
+ std::lock_guard lock(_block_mutex);
+ if (_block_pool.empty()) {
+ return std::make_unique<vectorized::Block>(tuple_desc()->slots(), 4);
+ }
+ auto block = std::move(_block_pool.back());
+ _block_pool.pop_back();
+ return block;
+}
+
+void Reusable::return_block(std::unique_ptr<vectorized::Block>& block) {
+ std::lock_guard lock(_block_mutex);
+ if (_block_pool.size() > 128) {
+ return;
+ }
+ block->clear_column_data();
+ _block_pool.push_back(std::move(block));
+}
+
+Status TabletLookupMetric::init(const PTabletKeyLookupRequest* request,
+ PTabletKeyLookupResponse* response) {
+ SCOPED_TIMER(&_profile_metrics.init_ns);
+ _response = response;
+ // using cache
+ uint128 uuid {static_cast<uint64_t>(request->uuid().uuid_high()),
+ static_cast<uint64_t>(request->uuid().uuid_low())};
+ auto cache_handle = LookupCache::instance().get(uuid);
+ _binary_row_format = request->is_binary_row();
+ if (cache_handle != nullptr) {
+ _reusable = cache_handle;
+ _hit_lookup_cache = true;
+ } else {
+ // init handle
+ auto reusable_ptr = std::make_shared<Reusable>();
+ TDescriptorTable t_desc_tbl;
+ TExprList t_output_exprs;
+ uint32_t len = request->desc_tbl().size();
+ RETURN_IF_ERROR(
+ deserialize_thrift_msg(reinterpret_cast<const uint8_t*>(request->desc_tbl().data()),
+ &len, false, &t_desc_tbl));
+ len = request->output_expr().size();
+ RETURN_IF_ERROR(deserialize_thrift_msg(
+ reinterpret_cast<const uint8_t*>(request->output_expr().data()), &len, false,
+ &t_output_exprs));
+ _reusable = reusable_ptr;
+ if (uuid != 0) {
+ LookupCache::instance().add(uuid, reusable_ptr);
+ // could be reused by requests after, pre allocte more blocks
+ RETURN_IF_ERROR(reusable_ptr->init(t_desc_tbl, t_output_exprs.exprs, 128));
+ } else {
+ RETURN_IF_ERROR(reusable_ptr->init(t_desc_tbl, t_output_exprs.exprs, 1));
+ }
+ }
+ _tablet = StorageEngine::instance()->tablet_manager()->get_tablet(request->tablet_id());
+ if (_tablet == nullptr) {
+ LOG(WARNING) << "failed to do tablet_fetch_data. tablet [" << request->tablet_id()
+ << "] is not exist";
+ return Status::NotFound(fmt::format("tablet {} not exist", request->tablet_id()));
+ }
+ RETURN_IF_ERROR(_init_keys(request));
+ _result_block = _reusable->get_block();
+ DCHECK(_result_block != nullptr);
+ return Status::OK();
+}
+
+Status TabletLookupMetric::lookup_up() {
+ RETURN_IF_ERROR(_lookup_row_key());
+ RETURN_IF_ERROR(_lookup_row_data());
+ RETURN_IF_ERROR(_output_data());
+ return Status::OK();
+}
+
+std::string TabletLookupMetric::print_profile() {
+ auto init_us = _profile_metrics.init_ns.value() / 1000;
+ auto init_key_us = _profile_metrics.init_key_ns.value() / 1000;
+ auto lookup_key_us = _profile_metrics.lookup_key_ns.value() / 1000;
+ auto lookup_data_us = _profile_metrics.lookup_data_ns.value() / 1000;
+ auto output_data_us = _profile_metrics.output_data_ns.value() / 1000;
+ auto total_us = init_us + lookup_key_us + lookup_data_us + output_data_us;
+ return fmt::format(
+ ""
+ "[lookup profile:{}us] init:{}us, init_key:{}us,"
+ ""
+ ""
+ "lookup_key:{}us, lookup_data:{}us, output_data:{}us, hit_lookup_cache:{}"
+ ""
+ ""
+ ", is_binary_row:{}, output_columns:{}"
+ "",
+ total_us, init_us, init_key_us, lookup_key_us, lookup_data_us, output_data_us,
+ _hit_lookup_cache, _binary_row_format, _reusable->output_exprs().size());
+}
+
+Status TabletLookupMetric::_init_keys(const PTabletKeyLookupRequest* request) {
+ SCOPED_TIMER(&_profile_metrics.init_key_ns);
+ // 1. get primary key from conditions
+ std::vector<OlapTuple> olap_tuples;
+ olap_tuples.resize(request->key_tuples().size());
+ for (size_t i = 0; i < request->key_tuples().size(); ++i) {
+ const KeyTuple& key_tuple = request->key_tuples(i);
+ for (const std::string& key_col : key_tuple.key_column_rep()) {
+ olap_tuples[i].add_value(key_col);
+ }
+ }
+ _primary_keys.resize(olap_tuples.size());
+ // get row cursor and encode keys
+ for (size_t i = 0; i < olap_tuples.size(); ++i) {
+ RowCursor cursor;
+ RETURN_IF_ERROR(cursor.init_scan_key(_tablet->tablet_schema(), olap_tuples[i].values()));
+ RETURN_IF_ERROR(cursor.from_tuple(olap_tuples[i]));
+ encode_key_with_padding<RowCursor, true, true>(
+ &_primary_keys[i], cursor, _tablet->tablet_schema()->num_key_columns(), true);
+ }
+ return Status::OK();
+}
+
+Status TabletLookupMetric::_lookup_row_key() {
+ SCOPED_TIMER(&_profile_metrics.lookup_key_ns);
+ _row_locations.reserve(_primary_keys.size());
+ // 2. lookup row location
+ Status st;
+ for (size_t i = 0; i < _primary_keys.size(); ++i) {
+ RowLocation location;
+ st = (_tablet->lookup_row_key(_primary_keys[i], nullptr, &location,
+ INT32_MAX /*rethink?*/));
+ if (st.is_not_found()) {
+ continue;
Review Comment:
I will add it in next PR, since my next PR modified related issue
##########
be/src/runtime/datetime_value.h:
##########
@@ -588,11 +588,13 @@ class DateTimeValue {
template <typename T>
void convert_from_date_v2(doris::vectorized::DateV2Value<T>* dt) {
+ this->_microsecond = 0;
Review Comment:
DateTimeValue lost _microsecond in this function, it's an old doris DateTime format so it does not need `_microsecond ` precision, but in this PR mysql_row_buffer need this precision to keep code compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
xiaokang commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1062070504
##########
be/src/util/mysql_global.h:
##########
@@ -26,13 +26,16 @@ typedef unsigned char uchar;
#define int1store(T, A) *((uint8_t*)(T)) = (uint8_t)(A)
#define int2store(T, A) *((uint16_t*)(T)) = (uint16_t)(A)
+#define int4store(T, A) *((uint32_t*)(T)) = (uint32_t)(A)
#define int3store(T, A) \
do { \
*(T) = (uchar)((A)); \
*(T + 1) = (uchar)(((uint32_t)(A) >> 8)); \
*(T + 2) = (uchar)(((A) >> 16)); \
} while (0)
#define int8store(T, A) *((int64_t*)(T)) = (uint64_t)(A)
+#define float4store(T, A) memcpy(T, (&A), sizeof(float))
Review Comment:
Why is float not the same as int?
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (type == TYPE_HLL) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (is_string_type(type)) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->insert_data(blob->getBlob(), blob->getBlobLen());
+ } else {
+ switch (type) {
+ case TYPE_BOOLEAN: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_TINYINT: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_SMALLINT: {
+ assert(slot_value->isInt16());
+ dst->insert(static_cast<JsonbInt16Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_INT: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_BIGINT: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_LARGEINT: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE: {
+ assert(slot_value->isDouble());
+ dst->insert(static_cast<JsonbDoubleVal*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATE: {
+ assert(slot_value->isInt32());
+ int32_t val = static_cast<JsonbInt32Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATETIME: {
+ assert(slot_value->isInt64());
+ int64_t val = static_cast<JsonbInt64Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATEV2: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATETIMEV2: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL32: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL64: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL128I: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ default:
+ LOG(FATAL) << "unknow type " << type;
+ break;
+ }
+ }
+}
+
+// column value -> jsonb
+static void serialize_column(Arena* mem_pool, const TabletColumn& tablet_column,
+ const IColumn* column, const StringRef& data_ref, int row,
+ JsonbWriterT<JsonbOutStream>& jsonb_writer) {
+ jsonb_writer.writeKey(tablet_column.unique_id());
+ if (is_column_null_at(row, column, tablet_column.type(), data_ref)) {
+ jsonb_writer.writeNull();
+ return;
+ }
+ if (tablet_column.is_array_type()) {
+ const char* begin = nullptr;
+ StringRef value = column->serialize_value_into_arena(row, *mem_pool, begin);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(value.data, value.size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_OBJECT) {
+ auto bitmap_value = (BitmapValue*)(data_ref.data);
+ auto size = bitmap_value->getSizeInBytes();
+ // serialize the content of string
+ auto ptr = mem_pool->alloc(size);
+ bitmap_value->write(reinterpret_cast<char*>(ptr));
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_HLL) {
+ auto hll_value = (HyperLogLog*)(data_ref.data);
+ auto size = hll_value->max_serialized_size();
+ auto ptr = reinterpret_cast<char*>(mem_pool->alloc(size));
+ size_t actual_size = hll_value->serialize((uint8_t*)ptr);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), actual_size);
+ jsonb_writer.writeEndBinary();
+ } else if (is_jsonb_blob_type(tablet_column.type())) {
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(data_ref.data), data_ref.size);
+ jsonb_writer.writeEndBinary();
+ } else {
+ switch (tablet_column.type()) {
+ case OLAP_FIELD_TYPE_BOOL: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_TINYINT: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_SMALLINT: {
+ int16_t val = *reinterpret_cast<const int16_t*>(data_ref.data);
+ jsonb_writer.writeInt16(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_INT: {
+ int32_t val = *reinterpret_cast<const int32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_BIGINT: {
+ int64_t val = *reinterpret_cast<const int64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_LARGEINT: {
+ __int128_t val = *reinterpret_cast<const __int128_t*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_FLOAT: {
+ float val = *reinterpret_cast<const float*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DOUBLE: {
+ double val = *reinterpret_cast<const double*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATE: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt32(datetime_cur->to_olap_date());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIME: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt64(datetime_cur->to_olap_datetime());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATEV2: {
+ uint32_t val = *reinterpret_cast<const uint32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIMEV2: {
+ uint64_t val = *reinterpret_cast<const uint64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL32: {
+ Decimal32::NativeType val =
+ *reinterpret_cast<const Decimal32::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL64: {
+ Decimal64::NativeType val =
+ *reinterpret_cast<const Decimal64::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL128I: {
+ Decimal128I::NativeType val =
+ *reinterpret_cast<const Decimal128I::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL:
+ LOG(FATAL) << "OLAP_FIELD_TYPE_DECIMAL not implemented use DecimalV3 instead";
+ break;
+ default:
+ LOG(FATAL) << "unknow type " << tablet_column.type();
+ break;
+ }
+ }
+}
+
+void JsonbSerializeUtil::block_to_jsonb(const TabletSchema& schema, const Block& block,
+ ColumnString& dst, int num_cols) {
+ auto num_rows = block.rows();
+ Arena pool;
+ assert(num_cols <= block.columns());
+ for (int i = 0; i < num_rows; ++i) {
+ JsonbWriterT<JsonbOutStream> jsonb_writer;
+ jsonb_writer.writeStartObject();
+ for (int j = 0; j < num_cols; ++j) {
+ const auto& column = block.get_by_position(j).column;
+ const auto& tablet_column = schema.columns()[j];
+ const auto& data_ref =
+ !tablet_column.is_array_type() ? column->get_data_at(i) : StringRef();
Review Comment:
Why use empty StringRef for array?
##########
be/src/util/mysql_global.h:
##########
@@ -26,13 +26,16 @@ typedef unsigned char uchar;
#define int1store(T, A) *((uint8_t*)(T)) = (uint8_t)(A)
#define int2store(T, A) *((uint16_t*)(T)) = (uint16_t)(A)
+#define int4store(T, A) *((uint32_t*)(T)) = (uint32_t)(A)
Review Comment:
It's better to keep 1 2 3 4 order
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
Review Comment:
add impl
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (type == TYPE_HLL) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (is_string_type(type)) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->insert_data(blob->getBlob(), blob->getBlobLen());
+ } else {
+ switch (type) {
+ case TYPE_BOOLEAN: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_TINYINT: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_SMALLINT: {
+ assert(slot_value->isInt16());
+ dst->insert(static_cast<JsonbInt16Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_INT: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_BIGINT: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_LARGEINT: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_FLOAT:
Review Comment:
we can add float in JSONB to avoid precision problem
##########
be/src/service/tablet_lookup_metric.h:
##########
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "butil/containers/doubly_buffered_data.h"
+#include "common/status.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gutil/int128.h"
+#include "olap/tablet.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+
+// For caching point lookup pre allocted blocks and exprs
+class Reusable {
+public:
+ ~Reusable();
+
+ bool is_expired(int64_t ttl_ms) const {
+ return butil::gettimeofday_ms() - _create_timestamp > ttl_ms;
+ }
+
+ Status init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size = 1);
+
+ RuntimeState* runtime_state() { return _runtime_state.get(); }
+
+ std::unique_ptr<vectorized::Block> get_block();
+
+ // do not touch block after returned
+ void return_block(std::unique_ptr<vectorized::Block>& block);
+
+ TupleDescriptor* tuple_desc() { return _desc_tbl->get_tuple_descriptor(0); }
+
+ const std::vector<vectorized::VExprContext*>& output_exprs() { return _output_exprs_ctxs; }
+
+private:
+ // caching TupleDescriptor, output_expr, etc...
+ std::unique_ptr<RuntimeState> _runtime_state;
+ DescriptorTbl* _desc_tbl;
+ std::mutex _block_mutex;
+ // prevent from allocte too many tmp blocks
+ std::vector<std::unique_ptr<vectorized::Block>> _block_pool;
+ std::vector<vectorized::VExprContext*> _output_exprs_ctxs;
+ int64_t _create_timestamp = 0;
+};
+
+// A cache used for prepare stmt.
+// One connection per stmt perf uuid
+// Use DoublyBufferedData to wrap Cache for performance and thread safe,
+// since it's not barely modified
+class LookupCache {
+public:
+ // uuid to reusable
+ using Cache = phmap::flat_hash_map<uint128, std::shared_ptr<Reusable>>;
+ using CacheIter = Cache::iterator;
+
+ LookupCache() = default;
+ static LookupCache& instance() {
+ static LookupCache ins;
+ return ins;
+ }
+
+ void add(uint128 cache_id, std::shared_ptr<Reusable> item) {
+ assert(item != nullptr);
+ _double_buffer_cache.Modify(update_cache, std::make_pair(cache_id, item));
+ }
+
+ // find an item, return null if not exist
+ std::shared_ptr<Reusable> get(uint128 cache_id) {
+ butil::DoublyBufferedData<Cache>::ScopedPtr s;
+ if (_double_buffer_cache.Read(&s) != 0) {
+ LOG(WARNING) << "failed to get cache from double buffer data";
+ return nullptr;
+ }
+ auto it = s->find(cache_id);
+ if (it != s->end()) {
+ return it->second;
+ }
+ return nullptr;
+ }
+
+private:
+ butil::DoublyBufferedData<Cache> _double_buffer_cache;
+ // 30 seconds for expiring an item
+ int32_t _expir_seconds = config::tablet_lookup_cache_clean_interval;
+
+ static size_t update_cache(Cache& old_cache,
+ const std::pair<uint128, std::shared_ptr<Reusable>>& p) {
+ old_cache.emplace(p);
+ return 1;
+ }
+
+ static size_t remove_items(Cache& old_cache, const std::vector<uint128>& keys) {
+ for (size_t i = 0; i < keys.size(); ++i) {
+ old_cache.erase(keys[i]);
+ }
+ return 1;
+ }
+
+ // Called from StorageEngine::_start_clean_lookup_cache
+ friend class StorageEngine;
+ void prune() {
+ std::vector<uint128> expired_keys;
+ {
+ butil::DoublyBufferedData<Cache>::ScopedPtr s;
+ if (_double_buffer_cache.Read(&s) != 0) {
+ return;
+ }
+ for (auto it = s->begin(); it != s->end(); ++it) {
+ if (it->second->is_expired(_expir_seconds * 1000)) {
+ expired_keys.push_back(it->first);
+ }
+ }
+ }
+
+ _double_buffer_cache.Modify(remove_items, expired_keys);
+ LOG(INFO) << "prune lookup cache, total " << expired_keys.size() << " expired items";
+ }
+};
+
+struct Metrics {
+ Metrics()
+ : init_ns(TUnit::TIME_NS),
+ init_key_ns(TUnit::TIME_NS),
+ lookup_key_ns(TUnit::TIME_NS),
+ lookup_data_ns(TUnit::TIME_NS),
+ output_data_ns(TUnit::TIME_NS) {}
+ RuntimeProfile::Counter init_ns;
+ RuntimeProfile::Counter init_key_ns;
+ RuntimeProfile::Counter lookup_key_ns;
+ RuntimeProfile::Counter lookup_data_ns;
+ RuntimeProfile::Counter output_data_ns;
+};
+
+// An util to do tablet lookup
+class TabletLookupMetric {
Review Comment:
Is there any speical meaning for Metric in the class name?
##########
be/src/olap/tablet_schema.cpp:
##########
@@ -920,11 +923,25 @@ bool operator==(const TabletSchema& a, const TabletSchema& b) {
if (a._is_in_memory != b._is_in_memory) return false;
if (a._delete_sign_idx != b._delete_sign_idx) return false;
if (a._disable_auto_compaction != b._disable_auto_compaction) return false;
+ if (a._store_row_column != b._store_row_column) return false;
return true;
}
bool operator!=(const TabletSchema& a, const TabletSchema& b) {
return !(a == b);
}
+const TabletColumn& TabletSchema::row_oriented_column() {
+ static TabletColumn source_column(OLAP_FIELD_AGGREGATION_NONE,
Review Comment:
maybe can use a const field
##########
be/src/service/tablet_lookup_metric.cpp:
##########
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "service/tablet_lookup_metric.h"
+
+#include "olap/row_cursor.h"
+#include "olap/storage_engine.h"
+#include "service/internal_service.h"
+#include "util/defer_op.h"
+#include "util/key_util.h"
+#include "util/runtime_profile.h"
+#include "util/thrift_util.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vliteral.h"
+#include "vec/sink/vmysql_result_writer.cpp"
+
+namespace doris {
+
+Reusable::~Reusable() {
+ for (vectorized::VExprContext* ctx : _output_exprs_ctxs) {
+ ctx->close(_runtime_state.get());
+ }
+}
+
+Status Reusable::init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size) {
+ _runtime_state.reset(new RuntimeState());
+ RETURN_IF_ERROR(DescriptorTbl::create(_runtime_state->obj_pool(), t_desc_tbl, &_desc_tbl));
+ _runtime_state->set_desc_tbl(_desc_tbl);
+ _block_pool.resize(block_size);
+ for (int i = 0; i < _block_pool.size(); ++i) {
+ _block_pool[i] = std::make_unique<vectorized::Block>(tuple_desc()->slots(), 10);
+ }
+
+ RETURN_IF_ERROR(vectorized::VExpr::create_expr_trees(_runtime_state->obj_pool(), output_exprs,
+ &_output_exprs_ctxs));
+ RowDescriptor row_desc(tuple_desc(), false);
+ // Prepare the exprs to run.
+ RETURN_IF_ERROR(vectorized::VExpr::prepare(_output_exprs_ctxs, _runtime_state.get(), row_desc));
+ _create_timestamp = butil::gettimeofday_ms();
+ return Status::OK();
+}
+
+std::unique_ptr<vectorized::Block> Reusable::get_block() {
+ std::lock_guard lock(_block_mutex);
+ if (_block_pool.empty()) {
+ return std::make_unique<vectorized::Block>(tuple_desc()->slots(), 4);
+ }
+ auto block = std::move(_block_pool.back());
+ _block_pool.pop_back();
+ return block;
+}
+
+void Reusable::return_block(std::unique_ptr<vectorized::Block>& block) {
+ std::lock_guard lock(_block_mutex);
+ if (_block_pool.size() > 128) {
+ return;
+ }
+ block->clear_column_data();
+ _block_pool.push_back(std::move(block));
+}
+
+Status TabletLookupMetric::init(const PTabletKeyLookupRequest* request,
+ PTabletKeyLookupResponse* response) {
+ SCOPED_TIMER(&_profile_metrics.init_ns);
+ _response = response;
+ // using cache
+ uint128 uuid {static_cast<uint64_t>(request->uuid().uuid_high()),
+ static_cast<uint64_t>(request->uuid().uuid_low())};
+ auto cache_handle = LookupCache::instance().get(uuid);
+ _binary_row_format = request->is_binary_row();
+ if (cache_handle != nullptr) {
+ _reusable = cache_handle;
+ _hit_lookup_cache = true;
+ } else {
+ // init handle
+ auto reusable_ptr = std::make_shared<Reusable>();
+ TDescriptorTable t_desc_tbl;
+ TExprList t_output_exprs;
+ uint32_t len = request->desc_tbl().size();
+ RETURN_IF_ERROR(
+ deserialize_thrift_msg(reinterpret_cast<const uint8_t*>(request->desc_tbl().data()),
+ &len, false, &t_desc_tbl));
+ len = request->output_expr().size();
+ RETURN_IF_ERROR(deserialize_thrift_msg(
+ reinterpret_cast<const uint8_t*>(request->output_expr().data()), &len, false,
+ &t_output_exprs));
+ _reusable = reusable_ptr;
+ if (uuid != 0) {
+ LookupCache::instance().add(uuid, reusable_ptr);
+ // could be reused by requests after, pre allocte more blocks
+ RETURN_IF_ERROR(reusable_ptr->init(t_desc_tbl, t_output_exprs.exprs, 128));
+ } else {
+ RETURN_IF_ERROR(reusable_ptr->init(t_desc_tbl, t_output_exprs.exprs, 1));
+ }
+ }
+ _tablet = StorageEngine::instance()->tablet_manager()->get_tablet(request->tablet_id());
+ if (_tablet == nullptr) {
+ LOG(WARNING) << "failed to do tablet_fetch_data. tablet [" << request->tablet_id()
+ << "] is not exist";
+ return Status::NotFound(fmt::format("tablet {} not exist", request->tablet_id()));
+ }
+ RETURN_IF_ERROR(_init_keys(request));
+ _result_block = _reusable->get_block();
+ DCHECK(_result_block != nullptr);
+ return Status::OK();
+}
+
+Status TabletLookupMetric::lookup_up() {
+ RETURN_IF_ERROR(_lookup_row_key());
+ RETURN_IF_ERROR(_lookup_row_data());
+ RETURN_IF_ERROR(_output_data());
+ return Status::OK();
+}
+
+std::string TabletLookupMetric::print_profile() {
+ auto init_us = _profile_metrics.init_ns.value() / 1000;
+ auto init_key_us = _profile_metrics.init_key_ns.value() / 1000;
+ auto lookup_key_us = _profile_metrics.lookup_key_ns.value() / 1000;
+ auto lookup_data_us = _profile_metrics.lookup_data_ns.value() / 1000;
+ auto output_data_us = _profile_metrics.output_data_ns.value() / 1000;
+ auto total_us = init_us + lookup_key_us + lookup_data_us + output_data_us;
+ return fmt::format(
+ ""
+ "[lookup profile:{}us] init:{}us, init_key:{}us,"
+ ""
+ ""
+ "lookup_key:{}us, lookup_data:{}us, output_data:{}us, hit_lookup_cache:{}"
+ ""
+ ""
+ ", is_binary_row:{}, output_columns:{}"
+ "",
+ total_us, init_us, init_key_us, lookup_key_us, lookup_data_us, output_data_us,
+ _hit_lookup_cache, _binary_row_format, _reusable->output_exprs().size());
+}
+
+Status TabletLookupMetric::_init_keys(const PTabletKeyLookupRequest* request) {
+ SCOPED_TIMER(&_profile_metrics.init_key_ns);
+ // 1. get primary key from conditions
+ std::vector<OlapTuple> olap_tuples;
+ olap_tuples.resize(request->key_tuples().size());
+ for (size_t i = 0; i < request->key_tuples().size(); ++i) {
+ const KeyTuple& key_tuple = request->key_tuples(i);
+ for (const std::string& key_col : key_tuple.key_column_rep()) {
+ olap_tuples[i].add_value(key_col);
+ }
+ }
+ _primary_keys.resize(olap_tuples.size());
+ // get row cursor and encode keys
+ for (size_t i = 0; i < olap_tuples.size(); ++i) {
+ RowCursor cursor;
+ RETURN_IF_ERROR(cursor.init_scan_key(_tablet->tablet_schema(), olap_tuples[i].values()));
+ RETURN_IF_ERROR(cursor.from_tuple(olap_tuples[i]));
+ encode_key_with_padding<RowCursor, true, true>(
+ &_primary_keys[i], cursor, _tablet->tablet_schema()->num_key_columns(), true);
+ }
+ return Status::OK();
+}
+
+Status TabletLookupMetric::_lookup_row_key() {
+ SCOPED_TIMER(&_profile_metrics.lookup_key_ns);
+ _row_locations.reserve(_primary_keys.size());
+ // 2. lookup row location
+ Status st;
+ for (size_t i = 0; i < _primary_keys.size(); ++i) {
+ RowLocation location;
+ st = (_tablet->lookup_row_key(_primary_keys[i], nullptr, &location,
+ INT32_MAX /*rethink?*/));
+ if (st.is_not_found()) {
+ continue;
Review Comment:
maybe add a mark in _row_locations
##########
be/src/vec/CMakeLists.txt:
##########
@@ -55,6 +55,7 @@ set(VEC_FILES
columns/column_string.cpp
columns/column_vector.cpp
columns/columns_common.cpp
+ jsonb/serialize.cpp
Review Comment:
move to 'j' position
##########
be/src/util/mysql_row_buffer.cpp:
##########
@@ -290,7 +392,11 @@ int MysqlRowBuffer::push_double(double data) {
return 0;
}
-int MysqlRowBuffer::push_time(double data) {
+template <bool is_binary_format>
+int MysqlRowBuffer<is_binary_format>::push_time(double data) {
+ if constexpr (is_binary_format) {
+ LOG(FATAL) << "not implemented";
Review Comment:
add impl
##########
be/src/olap/rowset/segment_v2/segment_iterator.cpp:
##########
@@ -244,6 +244,7 @@ Status SegmentIterator::_get_row_ranges_by_keys() {
auto row_range = RowRanges::create_single(lower_rowid, upper_rowid);
RowRanges::ranges_union(result_ranges, row_range, &result_ranges);
}
+
Review Comment:
just blank line in this file diff
##########
be/src/olap/delta_writer.cpp:
##########
@@ -440,6 +440,7 @@ void DeltaWriter::_build_current_tablet_schema(int64_t index_id,
ptable_schema_param.indexes(i),
ori_tablet_schema);
}
+
Review Comment:
just blank line
##########
be/src/service/internal_service.cpp:
##########
@@ -391,6 +393,25 @@ void PInternalServiceImpl::fetch_table_schema(google::protobuf::RpcController* c
st.to_protobuf(result->mutable_status());
}
+Status PInternalServiceImpl::_tablet_fetch_data(const PTabletKeyLookupRequest* request,
+ PTabletKeyLookupResponse* response) {
+ TabletLookupMetric lookup_util;
+ RETURN_IF_ERROR(lookup_util.init(request, response));
+ RETURN_IF_ERROR(lookup_util.lookup_up());
+ LOG(INFO) << lookup_util.print_profile();
Review Comment:
is INFO suitable?
##########
be/src/service/tablet_lookup_metric.h:
##########
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "butil/containers/doubly_buffered_data.h"
+#include "common/status.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gutil/int128.h"
+#include "olap/tablet.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+
+// For caching point lookup pre allocted blocks and exprs
+class Reusable {
+public:
+ ~Reusable();
+
+ bool is_expired(int64_t ttl_ms) const {
+ return butil::gettimeofday_ms() - _create_timestamp > ttl_ms;
+ }
+
+ Status init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size = 1);
+
+ RuntimeState* runtime_state() { return _runtime_state.get(); }
+
+ std::unique_ptr<vectorized::Block> get_block();
+
+ // do not touch block after returned
+ void return_block(std::unique_ptr<vectorized::Block>& block);
+
+ TupleDescriptor* tuple_desc() { return _desc_tbl->get_tuple_descriptor(0); }
+
+ const std::vector<vectorized::VExprContext*>& output_exprs() { return _output_exprs_ctxs; }
+
+private:
+ // caching TupleDescriptor, output_expr, etc...
+ std::unique_ptr<RuntimeState> _runtime_state;
Review Comment:
Is it safe to destroy runtime_state at the end of Reusable's life cycle?
##########
be/src/vec/jsonb/serialize.cpp:
##########
@@ -0,0 +1,313 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/jsonb/serialize.h"
+
+#include "olap/hll.h"
+#include "olap/tablet_schema.h"
+#include "runtime/jsonb_value.h"
+#include "util/jsonb_stream.h"
+#include "util/jsonb_writer.h"
+#include "vec/common/arena.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+static inline bool is_column_null_at(int row, const IColumn* column, const doris::FieldType& type,
+ const StringRef& data_ref) {
+ if (type != OLAP_FIELD_TYPE_ARRAY) {
+ return data_ref.data == nullptr;
+ } else {
+ Field array;
+ column->get(row, array);
+ return array.is_null();
+ }
+}
+
+static bool is_jsonb_blob_type(FieldType type) {
+ return type == OLAP_FIELD_TYPE_VARCHAR || type == OLAP_FIELD_TYPE_CHAR ||
+ type == OLAP_FIELD_TYPE_STRING || type == OLAP_FIELD_TYPE_STRUCT ||
+ type == OLAP_FIELD_TYPE_ARRAY || type == OLAP_FIELD_TYPE_MAP ||
+ type == OLAP_FIELD_TYPE_HLL || type == OLAP_FIELD_TYPE_OBJECT ||
+ type == OLAP_FIELD_TYPE_JSONB;
+}
+
+// jsonb -> column value
+static void deserialize_column(PrimitiveType type, JsonbValue* slot_value, MutableColumnPtr& dst) {
+ if (type == TYPE_ARRAY) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->deserialize_and_insert_from_arena(blob->getBlob());
+ } else if (type == TYPE_OBJECT) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (type == TYPE_HLL) {
+ assert(slot_value->isBinary());
+ // TODO
+ } else if (is_string_type(type)) {
+ assert(slot_value->isBinary());
+ auto blob = static_cast<JsonbBlobVal*>(slot_value);
+ dst->insert_data(blob->getBlob(), blob->getBlobLen());
+ } else {
+ switch (type) {
+ case TYPE_BOOLEAN: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_TINYINT: {
+ assert(slot_value->isInt8());
+ dst->insert(static_cast<JsonbInt8Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_SMALLINT: {
+ assert(slot_value->isInt16());
+ dst->insert(static_cast<JsonbInt16Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_INT: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_BIGINT: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_LARGEINT: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_FLOAT:
+ case TYPE_DOUBLE: {
+ assert(slot_value->isDouble());
+ dst->insert(static_cast<JsonbDoubleVal*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATE: {
+ assert(slot_value->isInt32());
+ int32_t val = static_cast<JsonbInt32Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATETIME: {
+ assert(slot_value->isInt64());
+ int64_t val = static_cast<JsonbInt64Val*>(slot_value)->val();
+ dst->insert_many_fix_len_data(reinterpret_cast<const char*>(&val), 1);
+ break;
+ }
+ case TYPE_DATEV2: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DATETIMEV2: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL32: {
+ assert(slot_value->isInt32());
+ dst->insert(static_cast<JsonbInt32Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL64: {
+ assert(slot_value->isInt64());
+ dst->insert(static_cast<JsonbInt64Val*>(slot_value)->val());
+ break;
+ }
+ case TYPE_DECIMAL128I: {
+ assert(slot_value->isInt128());
+ dst->insert(static_cast<JsonbInt128Val*>(slot_value)->val());
+ break;
+ }
+ default:
+ LOG(FATAL) << "unknow type " << type;
+ break;
+ }
+ }
+}
+
+// column value -> jsonb
+static void serialize_column(Arena* mem_pool, const TabletColumn& tablet_column,
+ const IColumn* column, const StringRef& data_ref, int row,
+ JsonbWriterT<JsonbOutStream>& jsonb_writer) {
+ jsonb_writer.writeKey(tablet_column.unique_id());
+ if (is_column_null_at(row, column, tablet_column.type(), data_ref)) {
+ jsonb_writer.writeNull();
+ return;
+ }
+ if (tablet_column.is_array_type()) {
+ const char* begin = nullptr;
+ StringRef value = column->serialize_value_into_arena(row, *mem_pool, begin);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(value.data, value.size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_OBJECT) {
+ auto bitmap_value = (BitmapValue*)(data_ref.data);
+ auto size = bitmap_value->getSizeInBytes();
+ // serialize the content of string
+ auto ptr = mem_pool->alloc(size);
+ bitmap_value->write(reinterpret_cast<char*>(ptr));
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), size);
+ jsonb_writer.writeEndBinary();
+ } else if (tablet_column.type() == OLAP_FIELD_TYPE_HLL) {
+ auto hll_value = (HyperLogLog*)(data_ref.data);
+ auto size = hll_value->max_serialized_size();
+ auto ptr = reinterpret_cast<char*>(mem_pool->alloc(size));
+ size_t actual_size = hll_value->serialize((uint8_t*)ptr);
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(ptr), actual_size);
+ jsonb_writer.writeEndBinary();
+ } else if (is_jsonb_blob_type(tablet_column.type())) {
+ jsonb_writer.writeStartBinary();
+ jsonb_writer.writeBinary(reinterpret_cast<const char*>(data_ref.data), data_ref.size);
+ jsonb_writer.writeEndBinary();
+ } else {
+ switch (tablet_column.type()) {
+ case OLAP_FIELD_TYPE_BOOL: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_TINYINT: {
+ int8_t val = *reinterpret_cast<const int8_t*>(data_ref.data);
+ jsonb_writer.writeInt8(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_SMALLINT: {
+ int16_t val = *reinterpret_cast<const int16_t*>(data_ref.data);
+ jsonb_writer.writeInt16(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_INT: {
+ int32_t val = *reinterpret_cast<const int32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_BIGINT: {
+ int64_t val = *reinterpret_cast<const int64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_LARGEINT: {
+ __int128_t val = *reinterpret_cast<const __int128_t*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_FLOAT: {
+ float val = *reinterpret_cast<const float*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DOUBLE: {
+ double val = *reinterpret_cast<const double*>(data_ref.data);
+ jsonb_writer.writeDouble(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATE: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt32(datetime_cur->to_olap_date());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIME: {
+ const auto* datetime_cur = reinterpret_cast<const VecDateTimeValue*>(data_ref.data);
+ jsonb_writer.writeInt64(datetime_cur->to_olap_datetime());
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATEV2: {
+ uint32_t val = *reinterpret_cast<const uint32_t*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DATETIMEV2: {
+ uint64_t val = *reinterpret_cast<const uint64_t*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL32: {
+ Decimal32::NativeType val =
+ *reinterpret_cast<const Decimal32::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt32(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL64: {
+ Decimal64::NativeType val =
+ *reinterpret_cast<const Decimal64::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt64(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL128I: {
+ Decimal128I::NativeType val =
+ *reinterpret_cast<const Decimal128I::NativeType*>(data_ref.data);
+ jsonb_writer.writeInt128(val);
+ break;
+ }
+ case OLAP_FIELD_TYPE_DECIMAL:
+ LOG(FATAL) << "OLAP_FIELD_TYPE_DECIMAL not implemented use DecimalV3 instead";
+ break;
+ default:
+ LOG(FATAL) << "unknow type " << tablet_column.type();
+ break;
+ }
+ }
+}
+
+void JsonbSerializeUtil::block_to_jsonb(const TabletSchema& schema, const Block& block,
+ ColumnString& dst, int num_cols) {
+ auto num_rows = block.rows();
+ Arena pool;
+ assert(num_cols <= block.columns());
+ for (int i = 0; i < num_rows; ++i) {
+ JsonbWriterT<JsonbOutStream> jsonb_writer;
Review Comment:
writer can be reused and reset for each row
##########
be/src/service/tablet_lookup_metric.cpp:
##########
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "service/tablet_lookup_metric.h"
+
+#include "olap/row_cursor.h"
+#include "olap/storage_engine.h"
+#include "service/internal_service.h"
+#include "util/defer_op.h"
+#include "util/key_util.h"
+#include "util/runtime_profile.h"
+#include "util/thrift_util.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vliteral.h"
+#include "vec/sink/vmysql_result_writer.cpp"
+
+namespace doris {
+
+Reusable::~Reusable() {
+ for (vectorized::VExprContext* ctx : _output_exprs_ctxs) {
+ ctx->close(_runtime_state.get());
+ }
+}
+
+Status Reusable::init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size) {
+ _runtime_state.reset(new RuntimeState());
+ RETURN_IF_ERROR(DescriptorTbl::create(_runtime_state->obj_pool(), t_desc_tbl, &_desc_tbl));
+ _runtime_state->set_desc_tbl(_desc_tbl);
+ _block_pool.resize(block_size);
+ for (int i = 0; i < _block_pool.size(); ++i) {
+ _block_pool[i] = std::make_unique<vectorized::Block>(tuple_desc()->slots(), 10);
+ }
+
+ RETURN_IF_ERROR(vectorized::VExpr::create_expr_trees(_runtime_state->obj_pool(), output_exprs,
+ &_output_exprs_ctxs));
+ RowDescriptor row_desc(tuple_desc(), false);
+ // Prepare the exprs to run.
+ RETURN_IF_ERROR(vectorized::VExpr::prepare(_output_exprs_ctxs, _runtime_state.get(), row_desc));
+ _create_timestamp = butil::gettimeofday_ms();
+ return Status::OK();
+}
+
+std::unique_ptr<vectorized::Block> Reusable::get_block() {
+ std::lock_guard lock(_block_mutex);
+ if (_block_pool.empty()) {
+ return std::make_unique<vectorized::Block>(tuple_desc()->slots(), 4);
Review Comment:
magic number
##########
be/src/runtime/descriptors.h:
##########
@@ -302,6 +307,13 @@ class JdbcTableDescriptor : public TableDescriptor {
class TupleDescriptor {
public:
// virtual ~TupleDescriptor() {}
+ ~TupleDescriptor() {
Review Comment:
If we add explicit dtor, we should add explicit move ctor & assignment.
##########
be/src/olap/tablet.cpp:
##########
@@ -1844,6 +1846,103 @@ TabletSchemaSPtr Tablet::get_max_version_schema(std::lock_guard<std::shared_mute
return _max_version_schema;
}
+RowsetSharedPtr Tablet::get_rowset(const RowsetId& rowset_id) {
+ for (auto& version_rowset : _rs_version_map) {
+ if (version_rowset.second->rowset_id() == rowset_id) {
+ return version_rowset.second;
+ }
+ }
+ for (auto& stale_version_rowset : _stale_rs_version_map) {
+ if (stale_version_rowset.second->rowset_id() == rowset_id) {
+ return stale_version_rowset.second;
+ }
+ }
+ return nullptr;
+}
+
+Status Tablet::lookup_row_data(const RowLocation& row_location, const TupleDescriptor* desc,
+ vectorized::Block* block) {
+ // read row data
+ BetaRowsetSharedPtr rowset =
+ std::static_pointer_cast<BetaRowset>(get_rowset(row_location.rowset_id));
+ if (!rowset) {
+ return Status::NotFound(
+ fmt::format("rowset {} not found", row_location.rowset_id.to_string()));
+ }
+
+ const TabletSchemaSPtr tablet_schema = rowset->tablet_schema();
+ SegmentCacheHandle segment_cache;
+ RETURN_IF_ERROR(SegmentLoader::instance()->load_segments(rowset, &segment_cache, true));
+ // find segment
+ auto it = std::find_if(segment_cache.get_segments().begin(), segment_cache.get_segments().end(),
+ [&row_location](const segment_v2::SegmentSharedPtr& seg) {
+ return seg->id() == row_location.segment_id;
+ });
+ if (it == segment_cache.get_segments().end()) {
+ return Status::NotFound(fmt::format("rowset {} 's segemnt not found, seg_id {}",
+ row_location.rowset_id.to_string(),
+ row_location.segment_id));
+ }
+ // read from segment column by column, row by row
+ segment_v2::SegmentSharedPtr segment = *it;
+ size_t row_size = 0;
+ MonotonicStopWatch watch;
+ watch.start();
+ Defer _defer([&]() {
+ LOG_EVERY_N(INFO, 500) << "get a single_row, cost(us):" << watch.elapsed_time() / 1000
+ << ", row_size:" << row_size;
+ });
+ // TODO(lhy) too long, refacor
+ if (tablet_schema->store_row_column()) {
+ // create _source column
+ segment_v2::ColumnIterator* column_iterator = nullptr;
+ RETURN_IF_ERROR(segment->new_row_column_iterator(&column_iterator));
+ std::unique_ptr<segment_v2::ColumnIterator> ptr_guard(column_iterator);
+ segment_v2::ColumnIteratorOptions opt;
+ OlapReaderStatistics stats;
+ opt.file_reader = segment->file_reader().get();
+ opt.stats = &stats;
+ opt.use_page_cache = !config::disable_storage_page_cache;
+ column_iterator->init(opt);
+ // get and parse tuple row
+ vectorized::MutableColumnPtr column_ptr =
+ vectorized::DataTypeFactory::instance()
+ .create_data_type(TabletSchema::row_oriented_column())
+ ->create_column();
+ std::vector<segment_v2::rowid_t> rowids {
+ static_cast<segment_v2::rowid_t>(row_location.row_id)};
+ RETURN_IF_ERROR(column_iterator->read_by_rowids(rowids.data(), 1, column_ptr));
+ assert(column_ptr->size() == 1);
+ auto string_column = static_cast<vectorized::ColumnString*>(column_ptr.get());
+ vectorized::JsonbSerializeUtil::jsonb_to_block(*desc, *string_column, *block);
+ return Status::OK();
+ }
+ // read from segment column by column, row by row
+ for (int x = 0; x < desc->slots().size(); ++x) {
+ int index = tablet_schema->field_index(desc->slots()[x]->col_unique_id());
+ auto column = block->get_by_position(x).column->assume_mutable();
+ // TODO handle real default value
Review Comment:
what's real default value?
##########
be/src/service/tablet_lookup_metric.cpp:
##########
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "service/tablet_lookup_metric.h"
+
+#include "olap/row_cursor.h"
+#include "olap/storage_engine.h"
+#include "service/internal_service.h"
+#include "util/defer_op.h"
+#include "util/key_util.h"
+#include "util/runtime_profile.h"
+#include "util/thrift_util.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vliteral.h"
+#include "vec/sink/vmysql_result_writer.cpp"
+
+namespace doris {
+
+Reusable::~Reusable() {
+ for (vectorized::VExprContext* ctx : _output_exprs_ctxs) {
+ ctx->close(_runtime_state.get());
+ }
+}
+
+Status Reusable::init(const TDescriptorTable& t_desc_tbl, const std::vector<TExpr>& output_exprs,
+ size_t block_size) {
+ _runtime_state.reset(new RuntimeState());
+ RETURN_IF_ERROR(DescriptorTbl::create(_runtime_state->obj_pool(), t_desc_tbl, &_desc_tbl));
+ _runtime_state->set_desc_tbl(_desc_tbl);
+ _block_pool.resize(block_size);
+ for (int i = 0; i < _block_pool.size(); ++i) {
+ _block_pool[i] = std::make_unique<vectorized::Block>(tuple_desc()->slots(), 10);
Review Comment:
magic number 10
##########
be/src/runtime/datetime_value.h:
##########
@@ -588,11 +588,13 @@ class DateTimeValue {
template <typename T>
void convert_from_date_v2(doris::vectorized::DateV2Value<T>* dt) {
+ this->_microsecond = 0;
Review Comment:
Is it a bug of convert_from_date_v2?
##########
be/src/olap/tablet.cpp:
##########
@@ -1844,6 +1846,103 @@ TabletSchemaSPtr Tablet::get_max_version_schema(std::lock_guard<std::shared_mute
return _max_version_schema;
}
+RowsetSharedPtr Tablet::get_rowset(const RowsetId& rowset_id) {
+ for (auto& version_rowset : _rs_version_map) {
+ if (version_rowset.second->rowset_id() == rowset_id) {
+ return version_rowset.second;
+ }
+ }
+ for (auto& stale_version_rowset : _stale_rs_version_map) {
+ if (stale_version_rowset.second->rowset_id() == rowset_id) {
+ return stale_version_rowset.second;
+ }
+ }
+ return nullptr;
+}
+
+Status Tablet::lookup_row_data(const RowLocation& row_location, const TupleDescriptor* desc,
+ vectorized::Block* block) {
+ // read row data
+ BetaRowsetSharedPtr rowset =
+ std::static_pointer_cast<BetaRowset>(get_rowset(row_location.rowset_id));
+ if (!rowset) {
+ return Status::NotFound(
+ fmt::format("rowset {} not found", row_location.rowset_id.to_string()));
+ }
+
+ const TabletSchemaSPtr tablet_schema = rowset->tablet_schema();
+ SegmentCacheHandle segment_cache;
+ RETURN_IF_ERROR(SegmentLoader::instance()->load_segments(rowset, &segment_cache, true));
+ // find segment
+ auto it = std::find_if(segment_cache.get_segments().begin(), segment_cache.get_segments().end(),
+ [&row_location](const segment_v2::SegmentSharedPtr& seg) {
+ return seg->id() == row_location.segment_id;
+ });
+ if (it == segment_cache.get_segments().end()) {
+ return Status::NotFound(fmt::format("rowset {} 's segemnt not found, seg_id {}",
+ row_location.rowset_id.to_string(),
+ row_location.segment_id));
+ }
+ // read from segment column by column, row by row
+ segment_v2::SegmentSharedPtr segment = *it;
+ size_t row_size = 0;
+ MonotonicStopWatch watch;
+ watch.start();
+ Defer _defer([&]() {
+ LOG_EVERY_N(INFO, 500) << "get a single_row, cost(us):" << watch.elapsed_time() / 1000
+ << ", row_size:" << row_size;
+ });
+ // TODO(lhy) too long, refacor
+ if (tablet_schema->store_row_column()) {
+ // create _source column
+ segment_v2::ColumnIterator* column_iterator = nullptr;
+ RETURN_IF_ERROR(segment->new_row_column_iterator(&column_iterator));
+ std::unique_ptr<segment_v2::ColumnIterator> ptr_guard(column_iterator);
+ segment_v2::ColumnIteratorOptions opt;
+ OlapReaderStatistics stats;
+ opt.file_reader = segment->file_reader().get();
+ opt.stats = &stats;
+ opt.use_page_cache = !config::disable_storage_page_cache;
+ column_iterator->init(opt);
+ // get and parse tuple row
+ vectorized::MutableColumnPtr column_ptr =
+ vectorized::DataTypeFactory::instance()
+ .create_data_type(TabletSchema::row_oriented_column())
+ ->create_column();
+ std::vector<segment_v2::rowid_t> rowids {
+ static_cast<segment_v2::rowid_t>(row_location.row_id)};
+ RETURN_IF_ERROR(column_iterator->read_by_rowids(rowids.data(), 1, column_ptr));
+ assert(column_ptr->size() == 1);
+ auto string_column = static_cast<vectorized::ColumnString*>(column_ptr.get());
+ vectorized::JsonbSerializeUtil::jsonb_to_block(*desc, *string_column, *block);
+ return Status::OK();
+ }
+ // read from segment column by column, row by row
+ for (int x = 0; x < desc->slots().size(); ++x) {
Review Comment:
x -> i
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #15491: Pq opt
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1058970516
##########
be/src/olap/tablet_schema.cpp:
##########
@@ -920,11 +923,25 @@ bool operator==(const TabletSchema& a, const TabletSchema& b) {
if (a._is_in_memory != b._is_in_memory) return false;
if (a._delete_sign_idx != b._delete_sign_idx) return false;
if (a._disable_auto_compaction != b._disable_auto_compaction) return false;
+ if (a._store_row_column != b._store_row_column) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (a._store_row_column != b._store_row_column) { return false;
}
```
##########
be/src/service/tablet_lookup_metric.h:
##########
@@ -0,0 +1,183 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+
+#include "butil/containers/doubly_buffered_data.h"
+#include "common/status.h"
+#include "gen_cpp/internal_service.pb.h"
+#include "gutil/int128.h"
+#include "olap/tablet.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+
+// For caching point lookup pre allocted blocks and exprs
+class Reusable {
+public:
+ ~Reusable();
+
+ bool is_expired(int64_t ttl_ms) {
Review Comment:
warning: method 'is_expired' can be made const [readability-make-member-function-const]
```suggestion
bool is_expired(int64_t ttl_ms) const {
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,28 +171,34 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator==(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator!=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
Review Comment:
warning: function declared 'noreturn' should not return [clang-diagnostic-invalid-noreturn]
```cpp
}
^
```
##########
be/src/vec/sink/vmysql_result_writer.cpp:
##########
@@ -435,7 +421,17 @@ int VMysqlResultWriter::_add_one_cell(const ColumnPtr& column_ptr, size_t row_id
}
}
+<<<<<<< HEAD
Review Comment:
warning: version control conflict marker in file [clang-diagnostic-error]
```cpp
<<<<<<< HEAD
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
Review Comment:
warning: no member named 'set_unique_id' in 'doris::TabletColumn' [clang-diagnostic-error]
```cpp
c2.set_unique_id(2);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
+ pslot1.set_col_name("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ type_desc.to_protobuf(pslot1.mutable_slot_type());
+ pslot1.set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(pslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ PSlotDescriptor pslot2;
+ pslot2.set_col_name("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ type_desc2.to_protobuf(pslot2.mutable_slot_type());
+ pslot2.set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(pslot2);
+ read_desc.add_slot(slot2);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot2);
^
```
**be/src/runtime/descriptors.h:380:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
Review Comment:
warning: variable has incomplete type 'doris::PSlotDescriptor' [clang-diagnostic-error]
```cpp
PSlotDescriptor pslot1;
^
```
**be/src/runtime/descriptors.h:52:** forward declaration of 'doris::PSlotDescriptor'
```cpp
class PSlotDescriptor;
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
+ pslot1.set_col_name("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ type_desc.to_protobuf(pslot1.mutable_slot_type());
+ pslot1.set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(pslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ PSlotDescriptor pslot2;
+ pslot2.set_col_name("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ type_desc2.to_protobuf(pslot2.mutable_slot_type());
+ pslot2.set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(pslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128, 3, TYPE_DECIMAL128},
+ {"k4", OLAP_FIELD_TYPE_STRING, 4, TYPE_STRING},
+ {"k5", OLAP_FIELD_TYPE_DECIMAL128, 5, TYPE_DECIMAL128},
Review Comment:
warning: use of undeclared identifier 'OLAP_FIELD_TYPE_DECIMAL128'; did you mean 'OLAP_FIELD_TYPE_DECIMAL128I'? [clang-diagnostic-error]
```suggestion
{"k5", OLAP_FIELD_TYPE_DECIMAL128I, 5, TYPE_DECIMAL128},
```
**be/src/olap/olap_common.h:153:** 'OLAP_FIELD_TYPE_DECIMAL128I' declared here
```cpp
OLAP_FIELD_TYPE_DECIMAL128I = 33,
^
```
##########
be/src/vec/sink/vmysql_result_writer.h:
##########
@@ -30,8 +30,11 @@ class TFetchDataResult;
namespace vectorized {
class VExprContext;
+template <bool is_binary_format = false>
class VMysqlResultWriter final : public VResultWriter {
public:
+ typedef std::vector<std::unique_ptr<TFetchDataResult>> ResultList;
Review Comment:
warning: use 'using' instead of 'typedef' [modernize-use-using]
```suggestion
using ResultList = std::vector<std::unique_ptr<TFetchDataResult>>;
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,28 +171,34 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator==(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator!=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator<=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator>=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator>=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
Review Comment:
warning: function declared 'noreturn' should not return [clang-diagnostic-invalid-noreturn]
```cpp
}
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,28 +171,34 @@ class JsonbDocument {
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator==(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
Review Comment:
warning: function declared 'noreturn' should not return [clang-diagnostic-invalid-noreturn]
```cpp
}
^
```
##########
be/src/vec/sink/vmysql_result_writer.cpp:
##########
@@ -435,7 +421,17 @@
}
}
+<<<<<<< HEAD
Status VMysqlResultWriter::append_block(Block& input_block) {
Review Comment:
warning: 'VMysqlResultWriter' is not a class, namespace, or enumeration [clang-diagnostic-error]
```cpp
Status VMysqlResultWriter::append_block(Block& input_block) {
^
```
**be/src/vec/sink/vmysql_result_writer.h:33:** 'VMysqlResultWriter' declared here
```cpp
class VMysqlResultWriter final : public VResultWriter {
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,28 +171,34 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator==(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator!=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator<=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
Review Comment:
warning: function declared 'noreturn' should not return [clang-diagnostic-invalid-noreturn]
```cpp
}
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
Review Comment:
warning: no member named 'set_unique_id' in 'doris::TabletColumn' [clang-diagnostic-error]
```cpp
c1.set_unique_id(1);
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,28 +171,34 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator==(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator!=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator<=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator>=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator>=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator<(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator<(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
Review Comment:
warning: function declared 'noreturn' should not return [clang-diagnostic-invalid-noreturn]
```cpp
}
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
Review Comment:
warning: no member named 'set_type' in 'doris::TabletColumn' [clang-diagnostic-error]
```cpp
c1.set_type(OLAP_FIELD_TYPE_ARRAY);
^
```
##########
be/src/util/jsonb_document.h:
##########
@@ -168,28 +171,34 @@
const ObjectVal* operator->() const { return ((const ObjectVal*)payload_); }
public:
- bool operator==(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator==(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator!=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator!=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator<=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator<=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator>=(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator>=(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator<(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator<(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
- bool operator>(const JsonbDocument& other) const {
- LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ [[noreturn]] bool operator>(const JsonbDocument& other) const {
+ // LOG(FATAL) << "comparing between JsonbDocument is not supported";
+ assert(false);
}
Review Comment:
warning: function declared 'noreturn' should not return [clang-diagnostic-invalid-noreturn]
```cpp
}
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: invalid use of incomplete type 'doris::PTupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:51:** forward declaration of 'doris::PTupleDescriptor'
```cpp
class PTupleDescriptor;
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
+ pslot1.set_col_name("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ type_desc.to_protobuf(pslot1.mutable_slot_type());
+ pslot1.set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(pslot1);
+ read_desc.add_slot(slot);
Review Comment:
warning: 'add_slot' is a private member of 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
read_desc.add_slot(slot);
^
```
**be/src/runtime/descriptors.h:380:** declared private here
```cpp
void add_slot(SlotDescriptor* slot);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
Review Comment:
warning: calling a private constructor of class 'doris::TupleDescriptor' [clang-diagnostic-error]
```cpp
TupleDescriptor read_desc(PTupleDescriptor(), true);
^
```
**be/src/runtime/descriptors.h:378:** declared private here
```cpp
TupleDescriptor(const PTupleDescriptor& tdesc, bool own_slot = false);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
Review Comment:
warning: no member named 'set_type' in 'doris::TabletColumn' [clang-diagnostic-error]
```cpp
c2.set_type(OLAP_FIELD_TYPE_ARRAY);
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
+ pslot1.set_col_name("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ type_desc.to_protobuf(pslot1.mutable_slot_type());
+ pslot1.set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(pslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ PSlotDescriptor pslot2;
+ pslot2.set_col_name("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ type_desc2.to_protobuf(pslot2.mutable_slot_type());
+ pslot2.set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(pslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128, 3, TYPE_DECIMAL128},
Review Comment:
warning: use of undeclared identifier 'OLAP_FIELD_TYPE_DECIMAL128'; did you mean 'OLAP_FIELD_TYPE_DECIMAL128I'? [clang-diagnostic-error]
```suggestion
{"k3", OLAP_FIELD_TYPE_DECIMAL128I, 3, TYPE_DECIMAL128},
```
**be/src/olap/olap_common.h:153:** 'OLAP_FIELD_TYPE_DECIMAL128I' declared here
```cpp
OLAP_FIELD_TYPE_DECIMAL128I = 33,
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
+ pslot1.set_col_name("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ type_desc.to_protobuf(pslot1.mutable_slot_type());
+ pslot1.set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(pslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ PSlotDescriptor pslot2;
Review Comment:
warning: variable has incomplete type 'doris::PSlotDescriptor' [clang-diagnostic-error]
```cpp
PSlotDescriptor pslot2;
^
```
**be/src/runtime/descriptors.h:52:** forward declaration of 'doris::PSlotDescriptor'
```cpp
class PSlotDescriptor;
^
```
##########
be/test/vec/jsonb/serialize_test.cpp:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+#include <gtest/gtest.h>
+
+#include "vec/core/block.h"
+#include "vec/core/types.h"
+#define private public
+#include "olap/tablet_schema.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_decimal.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/columns/column_vector.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_bitmap.h"
+#include "vec/data_types/data_type_date.h"
+#include "vec/data_types/data_type_date_time.h"
+#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_nullable.h"
+#include "vec/data_types/data_type_number.h"
+#include "vec/data_types/data_type_string.h"
+#include "vec/jsonb/serialize.h"
+#include "vec/runtime/vdatetime_value.h"
+
+namespace doris::vectorized {
+
+void fill_block_with_array_int(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnVector<int32_t>::create();
+ // init column array with [[1,2,3],[],[4],[5,6]]
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 3, 3, 4, 6};
+ std::vector<int32_t> vals = {1, 2, 3, 4, 5, 6};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data((const char*)(&v), 0);
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeInt32>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_int(std::move(column_array_ptr), array_type,
+ "test_array_int");
+ block.insert(test_array_int);
+}
+
+void fill_block_with_array_string(vectorized::Block& block) {
+ auto off_column = vectorized::ColumnVector<vectorized::ColumnArray::Offset64>::create();
+ auto data_column = vectorized::ColumnString::create();
+ // init column array with [["abc","de"],["fg"],[], [""]];
+ std::vector<vectorized::ColumnArray::Offset64> offs = {0, 2, 3, 3, 4};
+ std::vector<std::string> vals = {"abc", "de", "fg", ""};
+ for (size_t i = 1; i < offs.size(); ++i) {
+ off_column->insert_data((const char*)(&offs[i]), 0);
+ }
+ for (auto& v : vals) {
+ data_column->insert_data(v.data(), v.size());
+ }
+
+ auto column_array_ptr =
+ vectorized::ColumnArray::create(std::move(data_column), std::move(off_column));
+ vectorized::DataTypePtr nested_type(std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr array_type(std::make_shared<vectorized::DataTypeArray>(nested_type));
+ vectorized::ColumnWithTypeAndName test_array_string(std::move(column_array_ptr), array_type,
+ "test_array_string");
+ block.insert(test_array_string);
+}
+
+TEST(BlockSerializeTest, Array) {
+ TabletSchema schema;
+ TabletColumn c1;
+ TabletColumn c2;
+ c1.set_name("k1");
+ c1.set_unique_id(1);
+ c1.set_type(OLAP_FIELD_TYPE_ARRAY);
+ c2.set_name("k2");
+ c2.set_unique_id(2);
+ c2.set_type(OLAP_FIELD_TYPE_ARRAY);
+ schema.append_column(c1);
+ schema.append_column(c2);
+ // array int and array string
+ vectorized::Block block;
+ fill_block_with_array_int(block);
+ fill_block_with_array_string(block);
+ MutableColumnPtr col = ColumnString::create();
+ // serialize
+ JsonbSerializeUtil::block_to_jsonb(schema, block, static_cast<ColumnString&>(*col.get()),
+ block.columns());
+ // deserialize
+ TupleDescriptor read_desc(PTupleDescriptor(), true);
+ // slot1
+ PSlotDescriptor pslot1;
+ pslot1.set_col_name("k1");
+ TypeDescriptor type_desc(TYPE_ARRAY);
+ type_desc.children.push_back(TypeDescriptor(TYPE_INT));
+ type_desc.to_protobuf(pslot1.mutable_slot_type());
+ pslot1.set_col_unique_id(1);
+ SlotDescriptor* slot = new SlotDescriptor(pslot1);
+ read_desc.add_slot(slot);
+
+ // slot2
+ PSlotDescriptor pslot2;
+ pslot2.set_col_name("k2");
+ TypeDescriptor type_desc2(TYPE_ARRAY);
+ type_desc2.children.push_back(TypeDescriptor(TYPE_STRING));
+ type_desc2.to_protobuf(pslot2.mutable_slot_type());
+ pslot2.set_col_unique_id(2);
+ SlotDescriptor* slot2 = new SlotDescriptor(pslot2);
+ read_desc.add_slot(slot2);
+
+ Block new_block = block.clone_empty();
+ JsonbSerializeUtil::jsonb_to_block(read_desc, static_cast<ColumnString&>(*col.get()),
+ new_block);
+ std::cout << block.dump_data() << std::endl;
+ std::cout << new_block.dump_data() << std::endl;
+ EXPECT_EQ(block.dump_data(), new_block.dump_data());
+}
+
+TEST(BlockSerializeTest, JsonbBlock) {
+ vectorized::Block block;
+ TabletSchema schema;
+ std::vector<std::tuple<std::string, FieldType, int, PrimitiveType>> cols {
+ {"k1", OLAP_FIELD_TYPE_INT, 1, TYPE_INT},
+ {"k2", OLAP_FIELD_TYPE_STRING, 2, TYPE_STRING},
+ {"k3", OLAP_FIELD_TYPE_DECIMAL128, 3, TYPE_DECIMAL128},
Review Comment:
warning: use of undeclared identifier 'TYPE_DECIMAL128'; did you mean 'TYPE_DECIMAL128I'? [clang-diagnostic-error]
```suggestion
{"k3", OLAP_FIELD_TYPE_DECIMAL128, 3, TYPE_DECIMAL128I},
```
**be/src/runtime/define_primitive_type.h:54:** 'TYPE_DECIMAL128I' declared here
```cpp
TYPE_DECIMAL128I, /* 30 */
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15491:
URL: https://github.com/apache/doris/pull/15491#issuecomment-1396781190
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071706562
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/LiteralExpr.java:
##########
@@ -262,4 +262,106 @@ public boolean isNullable() {
public void finalizeImplForNereids() throws AnalysisException {
}
+
+ // Parse from binary data, the format follows mysql binary protocal
+ // see https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_binary_resultset.html.
+ // Return next offset
+ public void setupParamFromBinary(ByteBuffer data) {
+ Preconditions.checkState(false,
+ "should implement this in derived class. " + this.type.toSql());
+ }
+
+ public static LiteralExpr getLiteralByMysqlType(int mysqlType) throws AnalysisException {
+ switch (mysqlType) {
+ // MYSQL_TYPE_TINY
+ case 1:
+ return LiteralExpr.create("0", Type.TINYINT);
+ // MYSQL_TYPE_SHORT
+ case 2:
+ return LiteralExpr.create("0", Type.SMALLINT);
+ // MYSQL_TYPE_LONG
+ case 3:
+ return LiteralExpr.create("0", Type.INT);
+ // MYSQL_TYPE_LONGLONG
+ case 8:
+ return LiteralExpr.create("0", Type.BIGINT);
+ // MYSQL_TYPE_FLOAT
+ case 4:
+ return LiteralExpr.create("0", Type.FLOAT);
+ // MYSQL_TYPE_DOUBLE
+ case 5:
+ return LiteralExpr.create("0", Type.DOUBLE);
+ // MYSQL_TYPE_DECIMAL
+ case 0:
+ // MYSQL_TYPE_NEWDECIMAL
+ case 246:
+ return LiteralExpr.create("0", Type.DECIMAL32);
+ // MYSQL_TYPE_TIME
+ case 11:
+ return LiteralExpr.create("", Type.TIME);
+ // MYSQL_TYPE_DATE
+ case 10:
+ return LiteralExpr.create("1970-01-01", Type.DATE);
Review Comment:
DATE or DATEV2 are all MYSQL_TYPE_DATE in mysql protocol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15491:
URL: https://github.com/apache/doris/pull/15491#issuecomment-1397942130
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on PR #15491:
URL: https://github.com/apache/doris/pull/15491#issuecomment-1383441653
@xiaokang PR modified according to code review, lets' continue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
xiaokang commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1062059550
##########
be/src/util/jsonb_stream.h:
##########
@@ -126,6 +126,11 @@ class JsonbOutStream : public std::ostream {
size_ += len;
}
+ void write(__int128 l) {
+ // TODO
Review Comment:
impl like int64_t
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071705709
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -3308,6 +3317,11 @@ public long getNextId() {
return idGenerator.getNextId();
}
+ // counter for prepared statement id
+ public long getNextStmtId() {
+ return this.stmtIdCounter.getAndIncrement();
Review Comment:
`stmtIdCounter` doesn't need to persist, since it' attached with a JDBC connect session
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071705978
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/PlaceHolderExpr.java:
##########
@@ -0,0 +1,180 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.NotImplementedException;
+import org.apache.doris.thrift.TExprNode;
+
+import com.google.common.base.Preconditions;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+// PlaceHolderExpr is a reference class point to real LiteralExpr
+public class PlaceHolderExpr extends LiteralExpr {
+ private static final Logger LOG = LogManager.getLogger(LiteralExpr.class);
+ private LiteralExpr lExpr;
+ int mysqlTypeCode = -1;
+
+ public PlaceHolderExpr() {
+
+ }
+
+ public void setTypeCode(int mysqlTypeCode) {
+ this.mysqlTypeCode = mysqlTypeCode;
+ }
+
+ protected PlaceHolderExpr(LiteralExpr literal) {
+ this.lExpr = literal;
+ }
+
+ protected PlaceHolderExpr(PlaceHolderExpr other) {
+ this.lExpr = other.lExpr;
+ }
+
+ public void setLiteral(LiteralExpr literal) {
+ this.lExpr = literal;
+ this.type = literal.getType();
+ }
+
+ public LiteralExpr createLiteralFromType() throws AnalysisException {
+ Preconditions.checkState(mysqlTypeCode > 0);
+ return LiteralExpr.getLiteralByMysqlType(mysqlTypeCode);
+ }
+
+ public static PlaceHolderExpr create(String value, Type type) throws AnalysisException {
+ Preconditions.checkArgument(!type.equals(Type.INVALID));
+ return new PlaceHolderExpr(LiteralExpr.create(value, type));
+ }
+
+ @Override
+ protected void toThrift(TExprNode msg) {
+ lExpr.toThrift(msg);
+ }
+
+ /*
+ * return real value
+ */
+ public Object getRealValue() {
+ // implemented: TINYINT/SMALLINT/INT/BIGINT/LARGEINT/DATE/DATETIME/CHAR/VARCHAR/BOOLEAN
+ Preconditions.checkState(false, "not implement this in derived class. " + this.type.toSql());
Review Comment:
yes, no used for now, just make compiler happy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] kpfly commented on pull request #15491: Pq opt
Posted by GitBox <gi...@apache.org>.
kpfly commented on PR #15491:
URL: https://github.com/apache/doris/pull/15491#issuecomment-1367339306
Add documents later pls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
dataroaring merged PR #15491:
URL: https://github.com/apache/doris/pull/15491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071705168
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -155,6 +157,7 @@ public void setUserQueryTimeout(long queryTimeout) {
}
private StatementContext statementContext;
+ private Map<String, PrepareStmtContext> preparedStmtCtxs = Maps.newHashMap();
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071705537
##########
fe/fe-core/src/main/java/org/apache/doris/planner/OriginalPlanner.java:
##########
@@ -338,7 +339,6 @@ private void pushSortToOlapScan() {
PlanNode node = fragment.getPlanRoot();
PlanNode parent = null;
- // OlapScanNode is the last node.
Review Comment:
it's a mistake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] eldenmoon commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
eldenmoon commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1071705168
##########
fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java:
##########
@@ -155,6 +157,7 @@ public void setUserQueryTimeout(long queryTimeout) {
}
private StatementContext statementContext;
+ private Map<String, PrepareStmtContext> preparedStmtCtxs = Maps.newHashMap();
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #15491: [Enhancement](point query optimize) improve performace of point query on primary keys
Posted by GitBox <gi...@apache.org>.
xiaokang commented on code in PR #15491:
URL: https://github.com/apache/doris/pull/15491#discussion_r1062060054
##########
be/src/util/jsonb_writer.h:
##########
@@ -234,6 +234,17 @@ class JsonbWriterT {
return 0;
}
+ uint32_t writeInt128(__int128_t v) {
+ if (!stack_.empty() && verifyValueState()) {
Review Comment:
add (first_ && stack_.empty()) check like writeInt64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org